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PREFACE 


This Fifth- Year Report from the Evaluation of the Comprehensive School Reform Program 
Implementation and Outcomes (ECSRIO) is the final report on the outcomes of the federally 
funded Comprehensive School Reform (CSR) program. It presents findings about the 
relationship between participation in the U.S. Department of Education’s Comprehensive School 
Reform program in 2002 and subsequent student achievement five years later. This report 
follows the study’s Third- Year Report which presented findings as of 2005 and was published in 
2008.! The Fifth- Year Report examines the CSR program throughout the country and its 
relationship with gains in student achievement as of 2007. 


The CSR program was established as a demonstration program in 1998 and authorized as a full 
program in 2002 as part of the No Child Left Behind Act (NCLB). It is one approach to help low- 
performing K-12 public schools meet state performance standards. The CSR program 
emphasizes two major concepts. First, the approach mandates that school reform should be 
comprehensive in nature, strengthening all aspects of school operations—curriculum, instruction, 
professional development, parental involvement and school organization. Second, the CSR 
approach should involve the use of scientifically based research models—that is, models with 
evidence of effectiveness in multiple settings. 


In 2001, the U.S. Department of Education contracted with WestEd and COSMOS Corporation 
to conduct a five-year study of the CSR program; ECSRIO, which involved a survey of 500 CSR 
schools and 500 comparison schools; case studies of 30 sites; and analyses of student 
achievement in all schools receiving CSR funding in 2002. This report presents the fifth-year 
findings of this study. 


1. The Second-Year and Fourth-Year reports are internal documents to the U.S. Department of Education. The 
Second-Year report is titled “Longitudinal Assessment of Comprehensive School Reform Program 
Implementation and Outcomes” and was compiled October 2004 by the U.S. Department of Education Office 
of the Under Secretary, Policy and Program Studies Service. The Fourth-Year report is titled “Evaluation of 
the Comprehensive School Reform Program Implementation and Outcomes (ECSRIO): Interim Internal 
Report Draft” and was compiled November 2007 by the U.S. Department of Education, Office of Planning, 
Evaluation, and Policy Development/Policy and Program Studies Service. 


Preface vii 


ACKNOWLEDGMENTS 


We wish to thank many individuals who contributed to this report. We are especially grateful to 
the state, district and school survey respondents as well as to all of the school or district staff 
members who planned or participated in site visits. We express our deep gratitude for your role 
in the study and the data you provided, without which this study would not have been possible. 


The information in this report was provided through the congressionally mandated Evaluation of 
the Comprehensive School Reform Program Implementation and Outcomes (ECSRIO), which 
was conducted by WestEd and COSMOS Corporation under contract to the U.S. Department of 
Education. Naida Tushnet of WestEd led this study from its inception in 2002 through the middle 
of 2007 while Robert Yin and, later, Emily Warner led the work of COSMOS Corporation on 
ECSRIO and Susan Cragle from DUERR Associates coordinated the survey data collection. 


Several individuals at the U.S. Department of Education provided report guidance and direction. 
We would like to acknowledge the assistance of Alan Ginsburg, director, Policy and Program 
Studies Service (PPSS), and David Goodwin, past director, Program and Analytic Studies 
Division, PPSS. In addition, Diane Crow, Menahem Herman, Sabrina Jones, Calvin Marshall, 
Hina Mohammed and Aaron Neumann in PPSS contributed to the direction of the report. 


We would like to acknowledge the contributions and guidance of the members of our Technical 
Working Group, including Carolyn Temple Adger, Geoffrey Borman, H. J. Green, Bryan Hassel, 
Elsie Leak, Valerie Lee, Paul Ruiz, Jean Rutherford, Malik Stewart, Sam Stringfield and Ken 
Wong. 


Although we appreciate the assistance and support of all of these individuals, any errors in 
judgment or fact are the responsibility of the authors. 


Acknowledgments 1x 


EXECUTIVE SUMMARY 


The Comprehensive School Reform (CSR) program was first established as a demonstration 
program in 1998. The program was subsequently authorized under the Elementary and 
Secondary Education Act of 1965, as amended by the No Child Left Behind Act of 2001 (NCLB). 
The program emphasizes two major concepts. First, the approach mandates that school reform 
should be comprehensive in nature, strengthening all aspects of school operations—curriculum, 
instruction, professional development, parental involvement and school organization. Second, 
the CSR approach involves the use of scientifically based research models—that is, models with 
evidence of effectiveness in multiple settings. 


In 2001, the U.S. Department of Education contracted with WestEd and COSMOS Corporation 
to conduct the Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO). This fifth-year final report of the study first updates the Third-Year Report 
(U.S. Department of Education, 2008) by addressing whether schools that received CSR awards 
had higher achievement gains and/or made greater progress implementing the 11 components 
than did non-CSR schools. Specifically, we address the following questions: 


> Was receipt of a CSR award associated with improvements in school-level 
mathematics and reading achievement five years later? 


>» Were schools that received CSR awards more likely five years later to implement 
the legislatively specified components of CSR than other schools? 


Next, this report’s final chapter describes and discusses what we learned over the course of the 
entire study about school reform by summarizing key findings across the major ECSRIO reports, 
including the recently completed examination of dramatic school improvement. This chapter also 
includes recommendations for future federal policy and research. 


BACKGROUND 


Originally funded as the Comprehensive School Reform Demonstration Program in 1998 with 
$145 million, the program became part of NCLB in 2002 with funding of $310 million. In FY 
2003 and FY 2004, Congress allocated $308 million for the CSR program, and in FY 2005, $205 
million. Between 1998 and 2006, nearly 7,000 schools nationwide received three-year awards to 
implement CSR models. In FY 2006, funding was appropriated only for a clearinghouse to 
support comprehensive school reform and not for school-based activities. 


NCLB defines the CSR program as containing 11 components, which are assumed to work 
together as schools undergo reform (Exhibit E.1). 
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Exhibit E.1 
Eleven Components of Comprehensive School Reform 
Described in the No Child Left Behind Act 


Proven methods and strategies for student learning, teaching, and school management that are based on 
scientifically based research and effective practices and that have been replicated successfully in schools with 
diverse characteristics. 


Comprehensive design for effective school functioning, integrating instruction, assessment, classroom 
management, and professional development and aligning these functions into a schoolwide reform plan 
designed to enable all students to meet challenging state content and performance standards and to address 
needs identified through a school needs assessment. 


Professional development, including high-quality and continuous teacher and staff professional development 
and training. 


Measurable goals for student performance and benchmarks for meeting those goals. 
Support from staff members, including school faculty, administrators and other staff members. 


Support for staff members, including school faculty, administrators and other staff members (added in 2001). 


Parent and community involvement, including meaningful involvement of parents and the local community in 
planning and implementing school improvement activities. 


External assistance, including high-quality external support and assistance from a comprehensive school 
reform entity (which may be a university) with experience in schoolwide reform and improvement. 


Evaluation, Including a plan to evaluate the implementation of school reforms and the student results achieved. 


. Coordination of resources, which involves identifying how other available resources (federal, state, local and 
private) help the school coordinate services to support and sustain the school reform. 


. Scientifically based research designed to significantly improve the academic achievement of students 
participating in such programs as compared with students in schools who have not participated in such programs 
or to provide strong evidence that such programs will significantly improve the academic achievement of 
participating children (added in 2001). 


Source: Title I, Part F, Section 1606 of the Elementary and Secondary Education Act of 1965, as amended by the No Child Left 
Behind Act of 2001.. 


ECSRIO, mandated by Section 1607 of the Elementary and Secondary Education Act, as 
amended by the No Child Left Behind Act, addresses several broad questions related to the CSR 
program. 


> How were CSR funds targeted? 
> How was reform implemented in CSR and non-CSR schools? 


> How was reform related to achievement outcomes? 


ECSRIO examines the implementation and outcomes of the cohort of CSR awardees that 
received their initial awards in 2002. Awards spanned three years, although some awardees 
carried over funds for an additional year. 


The series of studies conducted for the project view the CSR program through numerous lenses, 
including achievement analyses, surveys of school reform implementation, and multiple case 
studies. 


Three distinct methodological approaches were used: 
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> Multivariate statistical analyses comparing all Title I CSR with Title I non-CSR 
matched comparison schools to determine whether receiving a CSR award was 
related to school-level achievement increases three and five years later? 


> Quantitative descriptive analyses of CSR reform implementation from a survey of 
principals and teachers in a random sample of 436 CSR and 382 matched non- 
CSR comparison schools to determine the comprehensiveness of implementation 


> Qualitative case study analyses designed to study CSR reform implementation 
and illustrate the ways CSR components were implemented? as well as to 
understand the process by which chronically low-performing schools turned 
themselves around and sustained achievement gains* 


Only a small number of high schools received CSR awards in 2002. Consequently, the sample 
included too few high schools for analysis, so this report does not focus on them.> This study 
frequently found differences in outcomes between elementary and middle schools, including 
differences in the relationships between achievement and award receipt. Such differences are 
possibly related to the different structures in the two types of schools, with middle schools 
having subject-area departments and elementary schools mainly teaching students in self- 
contained classrooms. These structural differences may affect not only the nature of the reform 
approach each level of school adopts but also how well reforms are implemented across all 
grades, subjects and teachers. Consequently, the report includes separate findings for elementary 
and middle schools. 


It is important to note that ESCRIO is not a study of the implementation of individual CSR 
models and their relationships with achievement outcomes. Many other researchers have focused 
on individual models (see Borman, Hewes, Overman, & Brown, 2002, for a meta-analysis of 
many of these studies). Instead, ECSRIO is a study of average achievement gains in schools that 
receive CSR grants, irrespective of the models these schools chose. 


2. The achievement analyses rely on (a) standardized school-level achievement scores from the National 
Longitudinal School-Level State Assessment Score Database (NLSLSASD) maintained by the American 
Institutes for Research (AIR), (b) data supplied by the Education Data Exchange Network (EDEN) housed at 
the U.S. Department of Education, (c) the Council of Chief State School Officers (CCSSO), and (d) states’ 
Web sites. 

3. These case studies involved the study of CSR reform implementation in (a) 15 pairs of schools (15 CSR 
schools and 15 matched comparison schools) originally selected to be part of this study to illustrate the ways 
CSR components were implemented in both CSR and non-CSR schools and (b) case studies in 13 schools (9 
improving and 4 comparison schools) selected to extend the previous qualitative case study analyses of CSR 
and non-CSR schools. 

4. Results from the study of how underperforming schools were able to turn around their performance appear 
in the Achieving Dramatic School Improvement: An Exploratory Study report (U.S. Department of Education, 
2010). 

5. For more information on the number of high schools versus elementary and middle schools that received 
CSR awards, see Exhibit B-2 on page 57 of Longitudinal Assessment of Comprehensive School Reform 
Program Implementation and Outcomes: First-Year Report (U.S. Department of Education, 2004). 
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STUDY LIMITATIONS 


The multi-method approach to this evaluation, like most studies of practice (Lipsey, 2003), has 
limitations. 


First, all achievement analyses rely on standardized school-level achievement scores from 
NLSLSASD, CCSSO, EDEN, and state Web sites. The use of school-level outcome measures 
presents methodological and psychometric challenges. 


¢ School-level scores are aggregate measures of individual student performance within a 
school and, as such, do not account for variation in academic performance among 
students within a group. 


¢ These school level analyses are not sensitive to the fact that performance may be 
contingent on both individual factors and factors related to group membership such as 
membership in different classrooms and school environments. The multileveled structure 
of data is not adequately addressed in aggregate analyses, which limits the precision of 
the estimates produced. 


Despite these weaknesses, designating the school as the unit of analysis and using aggregate 
outcome measures is deemed appropriate given the goals of the evaluation and the nature of the 
CSR program. The analyses rely on aggregate outcome measures that are both meaningful to 
schools and relevant to the current policy environment. School-level proficiency is the 
performance measure for which all schools are held accountable. Although the use of aggregate 
measures may compromise precision, its relevance for school officials and policy-makers offsets 
its limitations. Moreover, the CSR program was designed to stimulate comprehensive change 
through a set of coordinated reform actions that influence school operations across the whole 
school. In this context, it is appropriate to use the school as the unit of analysis and school-level 
achievement measures to examine how the CSR program influences achievement. 


A second limitation of this study is shared with other efforts to assess program effectiveness 
across states over time. Such evaluations must overcome the difficulty of using the existing state 
assessments, which are designed to provide information about students’ progress toward 
mastering the content established in each state’s standards. Consequently, assessments differ. In 
fact, neither the content nor the criteria for determining proficiency are the same from state to 
state. Also, standards, assessments and proficiency criteria often change, making scores within 
states difficult to compare over time. Finally, because of the structure of state assessments 
(especially before NCLB), we were unable to track cohorts of students because such data were 
rare before 2004—05. During the time before NCLB, most states assessed students once during 
the elementary school years (typically fourth grade) and once during the middle school years 
(typically eighth grade). We chose these grades to represent school-level achievement measures 
during the baseline years wherever possible because they were the only data available. 
Throughout the study, we continued to focus on these grades as surrogate measures for school- 
level achievement to ensure consistency in our measures; that is, we sought to ensure that 
students at similar levels of age-specific development were being compared with each other 
rather than attempt to average students across different stages of development in later years 
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(when more assessment data were available) and compare them with the original group of fourth- 
and eighth-grade students. 


School-level assessment scores were standardized within states for each year to enable us to 
analyze outcomes across states and assessment instruments.° The standardization provides a 
common metric for all achievement outcomes; however, it does not account for all sources of 
heterogeneity among states resulting from differences in assessments, the stringency of state 
proficiency standards, and the fact that the analyses rely on different groups of students in each 
year. 


Third, this study also may be limited by nonrandom attrition of a large number of schools from 
some analyses because of survey nonresponse, the fact that several states did not report all data 
to the Common Core of Data, and the removal of non-CSR schools that received CSR awards 
after 2002. The unrepresentative nature of missing data in the Year 5 sample and the 
inadvisability of imputing such data from Year 3 responses were factors precluding analyses in 
this report on the relationship between implementation and achievement. 


MAIN FINDINGS 


In this section, we address the questions of whether schools that received their initial CSR 
awards in 2002 realized achievement gains in mathematics and reading five years later, whether 
CSR schools made gains in implementing the federally mandated components of the CSR 
program, and whether CSR schools adopting curriculum models with a scientific research base 
were more likely to experience subsequent achievement gains. 


Was receipt of a CSR award associated with improvements in school-level mathematics 


and reading achievement? 


>» Five years after initially receiving their CSR awards, schools receiving awards 
did not demonstrate larger achievement growth than matched comparison 
schools not receiving CSR grants. In fact, achievement gains were nonexistent 
in CSR elementary schools, were marginally lower than their comparison 
schools in middle school mathematics, and were no different from their 
comparison schools in middle school reading. 


Achievement in CSR schools was largely unchanged in the five years after receiving a CSR 
award (Exhibit E.2). Comparison schools had higher gains in middle school mathematics and 
reading, although the differences between CSR and comparison schools were only marginally 
significant for mathematics achievement and insignificant for reading achievement. These 
findings are largely consistent with findings from this study’s Third Year Report (U.S. 
Department of Education, 2008). 


6. See Exhibit 2 on pages 9-16 in the Third Year Report (U.S. Department of Education, 2008) for a list of the 
measures used for each state for the years 1999-00 to 2004—05 of this study. This report updates those data 
with percent proficient in 2006-07; they were the only outcome measures available. 
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Exhibit E.2 
Average Gains in Percent Proficient or Higher From 2003 to 2007 for CSR and Matched 
Non-CSR Schools 


Average Gain in Percent Proficient 


Non-CSR 
Schools 


CSR Schools Difference 


Elementary 
Mathematics 
Elementary 
Reading 
Middle School 
Mathematics 
Middle School 
Reading 


n.s. n.s. n.s. 


n.s. n.s. n.s. 


2.8** 


n.s. 2.5" n.s. 


Exhibit highlights: Comprehensive School Reform (CSR) schools had a statistically significant 


increase between 2002-03 and 2006-07 in middle school mathematics of 0.2 percentage points. 
However, matched non-CSR schools had larger statistically significant increases in middle school 
mathematics and reading of 2.8 and 2.5 percentage points, respectively. Nevertheless, only middle 
school mathematics had a marginally statistically significant difference between CSR schools and 
their matched non-CSR schools. 


Source: CSR Awards Database; Common Core of Data (CCD); National Longitudinal School- 
Level State Assessment Score Database (NLSLSASD), Education Data Exchange Network 
(EDEN), Council of Chief State School Officers (CCSSO), and state Web sites. 


Note: “n.s.” indicates not statistically significant; + p< .10; * p< .05; ** p< .01. Tests for the 
statistical significance of achievement gains for CSR and non-CSR schools are ¢-tests to assess 
whether the value is different from zero. The differences between CSR and non-CSR schools are 
assessed through paired t-tests. Values for percent proficient or higher are computed by 
multiplying the average standard deviation of the achievement measures by the statistically 
significant estimates for the relationships between scientifically based research model adoption 
and school-level achievement. 


Were schools that received CSR awards more likely to implement the legislatively 
specified components of CSR than other schools? 


> No, consistent with earlier study findings, both CSR and non-CSR schools 
implemented a similar number of legislatively specified components in both 
2003 and 2007. The number of components rose slightly for CSR and non- 
CSR schools during this period at both the elementary and middle school 
levels. 


CSR awards were three-year grants intended to stimulate the implementation of the 11 
components identified in NCLB. However, by 2007, CSR schools, on average, implemented 
fewer than five components five years after receiving their CSR awards (Exhibit E.3). 
Furthermore, implementation of the number of components at the elementary school level was 
nearly identical in both CSR and non-CSR schools. In 2003, CSR elementary schools reported 
implementing slightly more components than non-CSR schools; this difference was statistically 
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significant. However, this difference was no longer statistically significant by 2007. No 
significant differences existed in either 2003 or 2007 in the number of reform components 
implemented by CSR and non-CSR middle schools. Both CSR and non-CSR schools increased 
the number of reform components implemented from 2003 to 2007. 


Exhibit E. 3 
Average Number of CSR Components Implemented by CSR 
and Non-CSR Schools in 2003 and 2007 


Average Number of Components Implemented 
Elementary School Middle School 
Change Change 
from 2003 from 2003 
School Type 2003 2007 to 2007 2007 to 2007 


fesschools ——SC=~C~srSCi Tw | Pe | 


Difference Between CSR 


Exhibit highlights: In 2003, Comprehensive School Reform (CSR) elementary schools 
reported implementing a somewhat higher average number of components than non-CSR 
schools. No other statistically significant differences between CSR and non-CSR schools 
exist. Both CSR and non-CSR schools reported similar increases between 2003 and 2007 in 
the number of components implemented. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) surveys. 


Note: + p < .10; * p< .05; ** p< .01. N= 150 for CSR elementary schools; N = 124 for non- 
CSR elementary schools; VN = 112 for CSR and non-CSR middle schools. 


Adoption of Scientifically Based Research Models and Achievement Outcomes 


In the Third- Year Report, we reported on whether schools that received CSR awards adopted 
models with a scientific research base and whether that adoption was associated with 
achievement gains after three years.’ Those ratings derived from the Comprehensive School 
Reform Quality (CSRQ) Center ratings’ of the scientific research base in commonly adopted 
reform models to assess the extent to which the use of CSR approaches with such a base was 
associated with higher achievement. We found that only one-third of CSR schools adopted 
models that later received ratings from CSRQ that indicated they had a scientific research base. 
Most schools chose models that were not rated by CSRQ. 


7. See U.S. Department of Education (2008). 

8. The CSRQ Center ratings provide a scale for the breadth and quality of the research base for the 31 most 
widely adopted CSR models in elementary, middle and high schools. The CSRQ Center conducted a total of 
40 ratings because some models were rated at both the elementary and secondary levels. The categories are 
Very Strong, Moderately Strong, Moderate, Limited, Zero and Negative. The CSRQ Center was funded under 
the U.S. Department of Education’s Comprehensive School Reform Quality Initiatives program. 
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In general, schools that adopted comprehensive school reform models with scientific research 
bases had higher mathematics achievement gains compared with reading gains after three years. 
Mathematics achievement improved in low-performing elementary schools? and in all middle 
schools (including low-performing schools) that adopted models with scientific research bases. 
Mathematics achievement improved in low-performing CSR elementary schools that adopted 
models with scientific research bases, but it did not improve in non-low-performing CSR 
elementary schools (Exhibit E.4).!° Low-performing CSR elementary schools gained about 2 
percentage points in the percentage of students scoring at the proficient level or higher in 
mathematics if they adopted a model with a limited scientific research base and 4 percentage 
points if they adopted a model with a moderate or moderately strong scientific research base. 
There was no relationship between changes in elementary reading achievement and the selection 
of a model with a scientific research base. 


At the middle school level, all CSR schools (including low-performing schools) that adopted 
models with limited scientific research bases realized gains after three years in the percent 
proficient or higher levels (about 6 percentage points in mathematics and 3 percentage points in 
reading), although the relationship with reading achievement is only weakly significant.!! 
Furthermore, low-performing CSR middle schools that adopted a model with a moderate or 
higher research base had gains of about 4 percentage points in the percent proficient or higher 
levels in mathematics, although this relationship, too, is also only weakly significant. 


One possible explanation for these inconsistent findings lies with the way the CSRQ Center 
determined ratings. The CSRQ Center determined “strength of the research base” by the nature 
of the evaluation design, with more weight given to more rigorous designs and the number of 
models evaluated that use such designs. It is possible that some potent strategies had not been 
evaluated rigorously, so the ratings underplay their potential for effectiveness. It is also possible 
that implementation in schools in this study may have been less comprehensive than it was in 
schools included in the studies that the CSRQ Center used. Note that we did not update these 
results for this Fifth- Year Report because we had no way of knowing whether these schools 
continued to implement the models they chose five years earlier. 


9. Low-performing schools refer to those CSR schools that were in the lowest 25 percent of achievement of all 
CSR schools in 2001-02. 

10. The CSRQ rating scale includes Very Strong, Moderately Strong, Moderate, Limited, Zero and Negative 
categories. No model received a “Negative” or “Very Strong” rating. The analyses that examine the 
relationship between strength of the research base and achievement treat the scientific research base variable as 
a series of dummy coded variables for Limited and Moderate/Moderately Strong CSRQ ratings. 

11. Throughout this report, findings that are reported as weakly significant are statistically significant at p < 
10. 
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Exhibit E.4 
Average Gains in Percent Proficient or Higher From 2002 to 2005, by Subject Area and 
Strength of Scientific Research Base 


Average Gains in Percent Proficient or Higher 
Elementary School Middle School 
Mathematics Reading 


All CSR Schools 


Limited Scientific Research Base 3.36+ 
Moderate/Moderately Strong eer ee: eke ee 
Research Base 

Low-Performing CSR Schools 
Limited Scientific Research Base 


| 
Moderate/Moderately Strong 4.00* ae 416+ ae 
Research Base : _ , ox 


Exhibit highlights: At the elementary level, low-performing Comprehensive School Reform 
(CSR) schools that adopted models with limited or moderate to moderately strong scientific 
research bases had higher mathematics achievement; there was no relationship between reading 
achievement and models with a scientific research base. At the middle school level, CSR 
schools that adopted models with limited scientific research bases had higher mathematics and 
reading achievement gains in the percent proficient or higher levels than those that did not 
adopt such models; low-performing CSR schools that adopted models with moderate to 
moderately strong research bases had higher mathematics achievement than those that chose 
other models. 


Source: From “Comprehensive School Reform Quality (CSRQ) Center Report on Elementary 
School Comprehensive School Reform Models,” by CSRQ Center, 2005, Washington, DC: 
American Institutes for Research; “Comprehensive School Reform Quality (CSRQ) Center 
Report on Middle and High School Comprehensive School Reform Models,” by CSRQ Center, 
2006, Washington, DC: American Institutes for Research; CSR Awards Database; Common 
Core of Data (CCD); National Longitudinal School-Level State Assessment Score Database 
(NLSLSASD). 


Note: n.s. indicates not statistically significant; + p< .10; * p< .05; ** p< .01. Values for 
percent proficient or higher are computed by multiplying the average standard deviation of the 
achievement measures by the statistically significant estimates for the relationships between 
scientifically based research model adoption and school-level achievement. 


OVERALL STUDY CONCLUSIONS 


In this section, we present the key overall findings from the multiple studies forming the 
ESCRIO project as well as the major implications for future federal education policy-making and 
for conducting high-quality research and evaluation studies in this area. 
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Effectiveness of the Federal CSR Program 


The federal CSR program did not yield comprehensively reformed schools. Although states 
largely succeeded in providing CSR funds to those schools most in need, schools receiving CSR 
awards made little progress in implementing more than just a few of the legislatively mandated 
components and were largely indistinguishable from non-CSR schools that were similar in 
baseline achievement and demographics. 


Furthermore, the federal CSR program was not associated with widespread achievement gains. 
Although CSR schools did realize improvements in mathematics and reading achievement after 
three years of their initial awards, they improved at quite similar rates as those of comparable, 
non-CSR Title I schools. After five years, any gains that CSR schools made had evaporated so 
that mathematics and reading achievement were similar to baseline levels at the time they 
received their awards. 


There were some instances where we did observe achievement gains, however. These gains were 
somewhat more likely in mathematics in schools that selected models identified as having a 
scientific research base. However, only one-third of the schools receiving CSR awards selected 
reform models identified as having a scientific research basis. 


Examples of School Improvement 


Despite the finding that the federal CSR program as a whole was not associated with 
improvements in achievement or school operations, we did find instances where targeted, 
sustained efforts appeared to lead to achievement gains.!* Results from the Achieving Dramatic 
School Improvement: An Exploratory Study report (U.S. Department of Education, 2010) not 
only highlighted instances of noteworthy achievement gains among initially low-performing 
schools but also expanded the knowledge base about the challenges and intricacies of turning 
around achievement in these schools. In this exploratory set of case studies of rapid- 
improvement schools (i.e., initially low-performing schools that made substantial achievement 
gains in a short time period) and schools improving at a slower and steadier pace, we identified 
leadership, school climate, instruction and data use, and external support as important factors in 
the schools visited. At the same time, we observed that these factors were “put together” by 
schools in many different ways. Some, for example, placed greater emphasis on distributed 
leadership over transparent use of student-level data while others did the reverse. Further, the 
reform strategies interacted with each other and with their school’s preexisting context in 
multiple ways. We believe these cases add to a growing body of literature associating specific 
school-level policies and practices with school improvement while also demonstrating that there 
is no “one way” to implement an effective school improvement program because different 
factors play out in different ways in different settings. 


12. A substudy was undertaken of CSR schools that made dramatic improvements in achievement, focusing on 
factors such as the role of leadership, use of data, school climate, and external support in helping to turn around poor 
performance. While results are reported separately in the Achieving Dramatic School Improvement (U.S. 
Department of Education, 2010), we provide a summary on pages 51—56 of this report. 
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RECOMMENDATIONS FOR FEDERAL POLICY AND RESEARCH 


Given the limited effects of the CSR program, how can the federal government foster scalable or 
systemwide reform that resembles the successes observed in the case study sites? We would 
posit that with no “one way” to implement a successful school improvement program, federally 
supported improvement will require highly contextualized and resource intensive support. The 
U.S. Department of Education would need to begin to provide substantial amounts of direct and 
individually tailored technical assistance to state departments of education and possibly also to 
local school districts. The assistance the Department would provide to states could mirror the 
situationally dependent assistance some states in the case study sites provided to districts and 
schools. 


Future research for improving chronically low-performing schools could play a significant role 
in shaping the nature of not only subsequent federal efforts but also the efforts of other service 
providers so they are targeted on interventions most likely to result in positive effects. To do so, 
we recommend that a future research agenda address three critical needs. First, there is a need for 
a robust national database of school information—including achievement measures, school 
characteristics (e.g., demographics and staffing), budgets, and relevant reform activities—to be 
made publicly available. This recommendation supports the second recommendation—the study 
of school improvement in “real-time” instead of retrospective efforts. Third, the effective study 
of school reform relies on the effective measurement of reform activities. Well-developed school 
reform implementation measures should be both specific enough to effectively characterize the 
school reform effort and broad enough to allow for comparison across school contexts; research, 
then, can effectively identify that set of practices a school facing a particular set of circumstances 
can adopt and feel reasonably confident that those practices have been shown in previous 
research to yield positive reform. 
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l. INTRODUCTION 


The Comprehensive School Reform (CSR) program was first established as a demonstration 
program in 1998. The program was subsequently authorized under the Elementary and 
Secondary Education Act of 1965, as amended by the No Child Left Behind Act of 2001 (NCLB). 
It is one approach to help low-performing K—12 public schools meet state performance 
standards. The CSR program emphasizes two major concepts. First, the approach mandates that 
school reform should be comprehensive in nature, strengthening all aspects of school 
operations—curriculum, instruction, professional development, parental involvement and school 
organization. Second, the CSR approach involves the use of scientifically based research 
models—that is, models with evidence of effectiveness in multiple settings. 


Originally funded as the Comprehensive School Reform Demonstration (CSRD) Program in 
1998 with $145 million, the program became part of NCLB in 2002 with funding of $310 
million. In FY 2003 and FY 2004, Congress allocated $308 million for the CSR program, and in 
FY 2005, $205 million. Between 1998 and 2006, nearly 7,000 schools nationwide received 
three-year awards to implement CSR models. In FY 2006, funding was appropriated only for a 
clearinghouse to support comprehensive school reform and not for school-based activities. Since 
then, there have been no federal funds dedicated to schools under the CSR program.!3 


NCLB defines CSR as containing 11 components, which are assumed to work together as schools 
undergo reform (Exhibit 1). For example, the “support from staff members” component of the 
CSR program dictates that the entire school should adopt the reform model, while the “support 
for staff members” component focuses on helping teachers and other staff members learn to use 
the reform strategy. This report studies these components and their implementation. 


13. Title I schoolwide schools may use Title I funds to fund the adoption of CSR models. Furthermore, the 
clearinghouse has continued to receive federal funds. 
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Exhibit 1 
Eleven Components of Comprehensive School Reform 
Described in the No Child Left Behind Act 


Proven methods and strategies for student learning, teaching, and school management that are based on 
scientifically based research and effective practices and that have been replicated successfully in schools with 
diverse characteristics. 


Comprehensive design for effective school functioning, integrating instruction, assessment, classroom 
management, and professional development and aligning these functions into a schoolwide reform plan 
designed to enable all students to meet challenging state content and performance standards and to address 
needs identified through a school needs assessment. 


Professional development, including high-quality and continuous teacher and staff professional development 
and training. 


Measurable goals for student performance and benchmarks for meeting those goals. 
Support from staff members, including school faculty, administrators and other staff members. 


Support for staff members, including school faculty, administrators and other staff members (added in 2001). 


Parent and community involvement, including meaningful involvement of parents and the local community in 
planning and implementing school improvement activities. 


External assistance, including high-quality external support and assistance from a comprehensive school 
reform entity (which may be a university) with experience in schoolwide reform and improvement. 


Evaluation, Including a plan to evaluate the implementation of school reforms and the student results achieved. 


. Coordination of resources, which involves identifying how other available resources (federal, state, local and 
private) help the school coordinate services to support and sustain the school reform. 


. Scientifically based research designed to significantly improve the academic achievement of students 
participating in such programs as compared with students in schools who have not participated in such programs 
or to provide strong evidence that such programs will significantly improve the academic achievement of 
participating children (added in 2001). 


Source: Title I, Part F, Section 1606 of the Elementary and Secondary Education Act of 1965, as amended by the No Child Left 
Behind Act of 2001. 


In 2001, the U.S. Department of Education contracted with WestEd to conduct a study of the 
Federal CSR program. The Evaluation of the Comprehensive School Reform Program 
Implementation and Outcomes (ECSRIO), mandated by Section 1607 of the Elementary and 
Secondary Education Act, as amended by the No Child Left Behind Act, addresses four broad 
questions related to the CSR program. 


> How were CSR funds targeted? 
> How was reform implemented in CSR and non-CSR schools? 
> How did state and district conditions influence reform implementation? 


> How was reform related to achievement outcomes? 
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ECSRIO examines the implementation and outcomes of a cohort of CSR awardees that received 
their initial awards in 2002. Early findings in this study illustrated that states targeted CSR funds 
to low-performing, high-need schools. The Third-Year Report (U.S. Department of Education, 
2008) found that neither the receipt of CSR awards nor the implementation of CSR components 
were associated with achievement gains in mathematics or reading achievement. There were, 
however, mixed relationships between achievement and the adoption of models with scientific 
research bases. 


As policy-makers grappled with the increasing numbers of schools not making Adequate Yearly 
Progress (AYP) under NCLB, the U.S. Department of Education asked WestEd and the 
American Institutes for Research (AIR) to identify those CSR schools that made rapid 
improvement along with some making “slow and steady progress” during the study period and to 
“unpack” the efforts of those schools to understand what actions led to their improvements. In 
the 2007-08 school year, teams of researchers from WestEd and AIR visited 11 such schools and 
interviewed current and former school and district staff members, parents and community 
members. The results from these case studies can be found in the report Achieving Dramatic 
School Improvement: An Exploratory Study (U.S. Department of Education, 2010). 


This fifth-year, final report from the ECSRIO project serves two purposes. First, we update the 
third-year report by addressing whether schools that received CSR awards had higher 
achievement gains and/or made greater progress implementing the 11 components than did non- 
CSR schools. Specifically, we address the following questions: 


> Was receipt of a CSR award associated with improvements in school-level 
mathematics and reading achievement five years later? 


>» Were schools that received CSR awards more likely five years later to implement 
the legislatively specified components of CSR than other schools? 


The next chapter describes the data sources used for the Year 5 analyses. The subsequent two 
chapters report on the overall relationship between CSR award and achievement (Chapter ITI) 
and the comprehensiveness of CSR implementation (Chapter IV). These two chapters include 
discussions of methodological approaches because each question relies on different 
methodologies and on samples constructed in different manners. The report appendixes include 
the standard error tables (Appendix A) and data collection instruments (Appendix B). 


The second purpose of this report is to describe and discuss what we learned over the course of 
the entire study about school reform. The final chapter presents a summary of key findings from 
the entire ECSRIO study, focusing primarily on the lack of any relationship between the federal 
CSR program and achievement gains and specifically contrasting that finding with those from 
this project’s intensive case studies of dramatic school improvement and from the National 
Longitudinal Evaluation of Comprehensive School Reform (Aladjem et al., 2006), both of which 
point to instances where school reform efforts can be effective. This chapter also presents 
recommendations for improved federal policy and research support for school reform and 
improvement. 
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ll. DATA SOURCES 


This study compares implementation and school-level achievement between schools that 
received CSR awards and similar schools that did not receive such awards. Comparisons were 
done by using multiple research methods for collecting and organizing data and by incorporating 
information from teachers and principals about reform in their schools. Analyses included the 
following: 


> Multivariate statistical analyses comparing all Title I CSR with Title I non-CSR 
matched comparison schools to determine whether receiving a CSR award was 
related to school-level achievement increases over time!4 


> Quantitative descriptive analyses of CSR reform implementation from a survey of 
principals and teachers in a random sample of 436 CSR and 382 matched non- 
CSR comparison schools!> 


> Qualitative case study analyses of CSR reform implementation in 15 pairs of 
schools (15 CSR schools and 15 matched comparison schools) originally selected 
to be part of this study to illustrate the ways CSR components were implemented 
in both CSR and non-CSR schools and in 13 schools (9 improving and 4 
comparison schools) chosen to extend the previous qualitative case study analyses 
of CSR and non-CSR schools 


Using multiple methods increases the validity of key measures and results. In addition, this study 
is an evaluation of a program as it operates in the field. Such an evaluation cannot control the 
conditions under which activities take place. Consequently, the study cannot use an experimental 
design, and no causal claims can be made. The evaluation uses a quasi-experimental design, 
matching schools that received CSR funds with schools that did not receive funding, in both the 
quantitative and qualitative aspects of data collection and analysis. 


This section describes the samples and the following three data sources for ECSRIO: 
> School-level achievement measures 
> Surveys of school reform activities 
> Case studies of school reform activities 


Because each evaluation question relies on different methods and samples constructed in 
different ways, discussions of the specific methods used are included with the findings for each 
evaluation question. The last part of this section presents study limitations. 


14. The achievement analyses rely on standardized school-level achievement scores from the National 
Longitudinal School-Level State Assessment Score Database (NLSLSASD) maintained by AIR, data supplied 
by the Education Data Exchange Network (EDEN) housed at the U.S. Department of Education, the Council of 
Chief State School Officers (CCSSO), and states’ Websites. 

15. Most CSR awards were given to elementary and middle schools. Consequently, the sample included too 
few high schools for analysis. 
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SCHOOL-LEVEL ACHIEVEMENT MEASURES 


The achievement analyses rely on standardized school-level achievement scores from the 
National Longitudinal School-Level State Assessment Score Database (NLSLSASD) maintained 
by AIR, EDEN, CCSSO, and state Web sites. The NLSLSASD includes school-level measures 
of achievement for nearly all public schools. This database includes seven years of data, 
spanning 1998—99 through 2004—05; however, many states did not have or provide assessment 
data for the first year or two. Also, before 2003—04, nearly all states tested only a few grades. 
Therefore, for this study, one grade (typically fourth) was chosen to represent elementary 
achievement, and one grade (typically eighth) was chosen to represent middle school 
achievement in the school. School-level achievement data for 2006-07 were obtained from 
EDEN at the U.S. Department of Education and were supplemented with data from School Data 
Direct and from individual states’ Web sites.'° During the time before NCLB, most states 
assessed students once during the elementary school years (typically fourth grade) and once 
during the middle school years (typically eighth grade). We chose these grades to represent 
school-level achievement measures during the baseline years wherever possible because they 
were the only data available. Throughout the study, we continued to focus on these grades as 
surrogate measures for school-level achievement to ensure consistency in our measures; that is, 
we sought to ensure that students at similar levels of age-specific development were being 
compared with each other,, rather than attempt to average students across different stages of 
development in later years (when more assessment data were available) and compare them with 
the original group of fourth- and eighth-grade students. 


In most instances, the average scale score or percent proficient or higher was used to calculate a 
standardized z-score within each state for each year.!” In several instances, however, other 
measures such as percentile ranks were the only available outcome measures, and in those 
situations, standardization was done in the following manner: 


1. In each year, appropriate assessment measures for mathematics and reading/language arts 
in elementary and middle grades were selected. 


2. In many cases, fourth-grade scores were used for elementary schools and eighth-grade 
scores for middle schools; however, where those scores were not available, other 
proximate grades were used. 


3. The schools in each state in each year were ranked according to their achievement 
measure. 


4. Percentile ranks were computed from these rankings. 


16. School Data Direct is compiled and supported by CCSSO. It can be accessed online at 
http://www.schooldatadirect.org/. 

17. The z-score is calculated by subtracting the population mean from an individual score and dividing it by the 
population standard deviation. 
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5. Percentile ranks were converted to normal curve equivalent scores. 


6. Normal curve equivalent scores were converted to z-scores, so each state was standardized 
with a 0 mean and a standard deviation of 1. 


Where no assessment scores were available for a state in an academic year, the standardized 
score was estimated by averaging the standardized score from the prior year and that of the year 
after. For instance, there were no assessment measures available for Idaho in 2004. The 
elementary mathematics score for each school in Idaho in 2004 was computed by averaging the 
2003 and 2005 standardized scores in elementary school mathematics for each school. 


Because the standardized scores are standard deviations, coefficients are comparable to effect 
sizes; that is, coefficients are interpreted as changes in standard deviations of the outcome 
measure. Coefficients that result from analyses can also be converted to more interpretable 
measures such as percent proficient or higher (Gill et al., 2005). For example, if the CSR 
program accounts for an increase in achievement scores of 0.5 standard deviations, this value can 
be multiplied by the average standard deviation of the percentage of students who achieve at the 
proficient level or higher across states to develop a rough estimate of the increase in achievement 
levels associated with the CSR approach. Exhibit 2 presents the standard deviations of 
elementary and middle school mathematics and reading achievement calculated from the 45 
states in which achievement data were expressed as a percentage of those achieving proficient or 
higher levels in 2007. The average standard deviation for each of the achievement measures in 
elementary school mathematics and reading is approximately 20 percent and, for middle school 
mathematics and reading, 24.6 percent. Therefore, a finding that the CSR program accounts for 
an increase in achievement scores of 0.5 standard deviations corresponds to an increase of about 
10 percent in elementary schools and 12.3 percent in middle schools. 
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Exhibit 2 
Standard Deviations of Percent Proficient or Higher for 2007 Elementary and 
Middle School Mathematics and Reading Achievement, by State 
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Exhibit highlights: The average standard deviation of all of the states was between 19.9 and 
24.6 across school level and mathematics and reading subject areas. 


Source: Education Data Exchange Network (EDEN), Council of Chief State School Officers 
(CCSSO), and state Web sites. 


SURVEY OF SCHOOL REFORM ACTIVITIES 


The implementation analyses rely on survey data from a random sample of CSR schools and 
matched comparison non-CSR schools developed at the outset of this study. The survey provides 
descriptive quantitative data on reform implementation—that is, the extent that schools 
implemented the 11 elements of the CSR program included in NCLB and the extent that schools 
engaged in other non-CSR reform-related activities. The survey also examines other factors that 
research has associated with successful reform such as school organization (e.g. instructors teach 
all subjects versus the same subject to most classes; common planning periods). 


The evaluation team ensured that the sample of schools in the ECSRIO survey represented the 
population of schools by randomly selecting a large enough number of CSR awardees for the 
survey sample. When the sample for the study was drawn at the beginning of this study, the 
initial CSR universe for calendar year 2002 comprised 38 states and 1,096 schools. The sample 
of 400 schools was about 36 percent of this universe, representing 35 states. The evaluation team 
then updated the sample based on new data, increasing the sample by 100 schools from the 15 
remaining states (further explained below). 


To select the sample, researchers obtained the most complete list of CSR schools from the U.S. 
Department of Education’s CSR Awards Database. The Department of Education required states 
to report their awardees to the CSR Awards Database. However, the database did not include all 
states from the year 2002 for two reasons. First, although data collection had been ongoing, at 
any given time the data may be incomplete because of delinquent reporting by state education 
agencies. Second, many states award CSR funds to schools biennially or irregularly. 
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Of the initial 15 states in the 2002 CSR universe that were missing from the original sample, 
three states were left out as a by-product of the random selection process. Five additional states 
reported data after the initial sample selection. The remaining seven states did not make any 
awards in 2002 but reported 2003 data. During the first year of the study, the Department of 
Education allocated additional funds to gather data from these 15 states. 


Researchers purposefully selected an additional 100 schools from a larger random sample of 
these missing states to ensure that the sample represented all states and school levels. Also, to 
measure the value added by CSR to Title I schoolwide grants, a larger comparison group of 
schools with non-CSR Title I schoolwide grants was required. Thus, in the second sample, the 
choice of comparison schools was limited to those classified as having Title I schoolwide grants 
in 2002. The resulting combined sample of 500 CSR schools makes up 37 percent of the 
eventual (now relatively stable) universe of 1,340 awardees of the 2002 cohort. Survey data from 
the original sample have been combined with the follow-up sample for analyses in this report. 
Only a small number of high schools received CSR awards in 2002. Consequently, the high 
school survey sample was too small for statistical analyses, so this report focuses on elementary 
and middle schools, but not high schools. 


Researchers used a two-step process to select potential matches for comparison with CSR 
schools. First, they created a school equivalency index for all schools in each state (where data 
were available). Second, they calculated a proximity score between each pair of schools within a 
state. Matching schools were selected that had the closest proximity on the index to CSR schools 
within the same district. 


A regression-based approach to weighting and combining background characteristics was used to 
construct the school equivalency index for each state. This method is a simplified version of the 
California School Characteristics Index (which was used by the Technical Design Group of the 
Advisory Committee for the Public Schools Accountability Act of 1999). Using the NLSLSASD, 
the team regressed measures of academic performance on measures of schoolwide participation 
in federally subsidized free or reduced-price lunch programs and on schoolwide counts of 
student ethnicity. The estimated coefficients led to a composite of background characteristics for 
each school. In short, each background characteristic was weighted by the amount it contributed 
to student performance. 


The team selected potential matches for each first-year CSR school based on a minimum 
distance criterion. This method was used in the majority of cases. However, alternative methods 
were used in two circumstances. First, in some districts (or states), either demographic or 
performance data were not available. Second, a suitable comparison non-CSR school was not 
available within the same district because the district was too small or all other comparable 
schools had previously participated in the CSR program. Each of these contingencies is 
discussed below. 


In some cases, not enough data were available to construct a school equivalency index. Where 
states or districts did not report free or reduced-price lunch or ethnicity, schools were ranked 
within districts using only achievement scores. Examples of states where these data were not 
available are Tennessee and Washington. Achievement data were missing for some schools or 
districts. The proximity scores in these cases were based on an unweighted composite of free or 
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reduced-price lunch and ethnicity. Finally, in cases where neither student performance nor 
demographic data were available, non-CSR comparison schools were matched by school grade 
span, size, and locale. 


In districts where a non-CSR comparison school could not be selected, the team searched for a 
suitable comparison in an adjacent district or similar locale. Because the school equivalency 
index included all public schools in the state, the proximity of any school within the state could 
be calculated. The same criteria were used for selection across districts when data were available. 
In cases where data were not available, the team used the same procedures that applied to 
selecting comparison schools within districts. 


Among the schools selected at the outset of this study to chart implementation over the life of the 
CSR awards and beyond, non-CSR schools in the sample were higher-performing than the CSR 
schools in the sample and among all newly-awarded CSR schools in 2002. The baseline 
achievement in the ECSRIO sample of elementary CSR schools was slightly more than 0.3 
standard deviations lower than their matched non-CSR comparisons while the differences were 
approximately 0.4 for middle schools (Exhibit 3). This difference may be related to the 
procedures used to select CSR award recipients. The baseline achievement for the sample of 
CSR schools was higher than the baseline achievement for the universe of CSR schools. This 
difference may be because of the purposeful selection of the additional 100 schools. 


Exhibit 3 
Average Baseline Standardized Achievement Measures of All 2002 CSR 
Schools and of the CSR and Non-CSR Schools in the ECSRIO Sample 


ECSRIO CSR ECSRIO Non-CSR 
Sample Sample 
err ny Universe(SD Schools(SD Schools(SD 


ee 


Elementary School Reading 


Midele School Mathematics | -0.90 ‘| oer ——=«dts~StCaT Cid 
Miele School Reading 


Exhibit highlights: The average baseline standardized achievement measures of the 
Evaluation of the Comprehensive School Reform Program Implementation and Outcomes 
(ECSRIO) Comprehensive School Reform (CSR) schools was slightly more than 0.3 standard 
deviations lower than the average baseline standardized achievement measures of the 
ECSRIO non-CS8R schools. 


Source: CSR Awards Database; National Longitudinal School-Level State Assessment Score 
Database (NLSLSASD). 


In their demographic compositions in 2007, the samples of ECSRIO CSR and non-CSR schools 
were similar to each other and to the universe of schools that received their CSR awards in 2002 
(Exhibit 4). The universe of CSR schools had a higher percentage of students who were eligible 
for free or reduced-price lunch than did the ECSRIO CSR or non-CSR schools. The universe of 
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CSR schools had a higher percentage of black students than either the ECSRIO CSR schools or 
the non-CSR schools. The percentage of Hispanic students was similar across all categories. 


Exhibit 4 
Average Demographic Measures in 2007 of All 2002 CSR Schools 
and of the CSR and Non-CSR Schools in the ECSRIO Sample 


Demographic Measures CSR Universe Sample Schools Sample Schools 
Hispanic (%) 
Type of School (%) 


Exhibit highlights: The average percentages of students receiving free or reduced-price lunch, 
average percentage of black students, and average percentage of Hispanic students in the 
Evaluation of the Comprehensive School Reform Program Implementation and Outcomes 
(ECSRIO) Comprehensive School Reform (CSR) schools in 2007 were about the same as in 
the ECSRIO non-CSR schools. 


Source: CSR Awards Database; Common Core of Data (CCD); National Longitudinal 
School-Level State Assessment Score Database (NLSLSASD). 


From the 1,000 schools originally selected for the survey sample, 961 agreed to participate in the 
study and were surveyed in 2003 and again in 2005. Principals and up to three teachers in each 
school were surveyed on various aspects of their respective school’s reform activities (see 
Appendix B for the survey instruments). The response rate for the first-year data collection 
(2003) was 90 percent for the 479 CSR schools and 82 percent for the 482 non-CSR schools. The 
response rate for the third-year data collection (2005) was 88 percent for CSR schools and 78 
percent for non-CSR schools. The response rate for surveyed staff members was 77 percent and 
64 percent for principals and teachers, respectively, in 2003 and 75 percent and 71 percent, 
respectively, in 2005. 


In spring 2007, the third round of ECSRIO surveys was administered to principals and teachers. 
Several districts declined to participate in this round of surveys, reducing the initial sample from 
961 to 879 CSR and matched non-CSR schools. In addition, 61 non-CSR schools that received 
CSR awards after 2002 were removed from the sample so that the comparison would be between 
schools that received a 2002 CSR award and those that had not received a CSR award at any 
time, further reducing the total sample size to 818 schools (436 CSR and 382 non-CSR schools). 
Principals and up to 15 teachers in each school were surveyed on various aspects of their 
respective school’s reform activities. The principal response rate was 65 percent and 66 percent 
for CSR and non-CSR schools, respectively. Because principals were instructed to provide up to 
15 teachers with surveys and the number of possible teacher responses from school to school 
varied considerably, a conventional response rate for teachers cannot be calculated. 
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Administration of the teacher survey yielded at least one teacher response from 87 percent of 
CSR schools and 83 percent of non-CSR schools. 


The survey instrument includes items that measure behaviors rather than attitudes and 
expectations. For example, respondents were asked about procedures that monitor student 
achievement during implementation rather than whether the reform strategy meets student needs. 
Although this approach is likely to increase the objectivity of the responses, self-reported 
responses may still have limitations such as the accuracy of respondent recall and the tendency 
for respondents to give socially desirable responses (Flower, 1995; Tourangeau, Rips, & Rasinki, 
2000). Survey findings were analyzed in conjunction with the results of the case studies to 
validate information on implementation from both sources. 


CASE STUDIES OF SCHOOL REFORM ACTIVITIES 


The case studies included in this evaluation provide in-depth insight into comprehensive school 
reform implementation processes and outcomes. They also provide clues for interpreting the 
survey data to determine whether self-reported data reflect inflated or conservative responses. 


Of the 1,000 schools selected for the survey sample, a subsample of 15 pairs of schools from 15 
districts across 14 states—with each pair containing a CSR school and a non-CSR school—was 
originally selected to provide case studies that supplement the quantitative analyses with 
qualitative understanding of implementation dynamics. For example, although almost all schools 
reported having a comprehensive school plan, the case studies revealed differences in how the 
plan was used, with some schools developing a plan as a ritual activity and others making it into 
a living document. With such a small number of case study schools, this sample does not 
represent either the geographic distribution or the distribution of school levels in the universe of 
CSR schools. However, the selection ensured that the case study sample included enough 
geographic and school-level diversity to reflect a variety of state policy environments, including 
schools in states with a long tradition of local control and others in states that exercise more 
centralized control over schools. 


Researchers selected CSR and comparison pairs that resided in the same district for the field- 
based study because it was the only way to observe the differential effect of district policies. 

Also, visiting a single district for each pair minimized the data collection burden for both the 
evaluators and respondents. 


The case studies included two visits to each site, with a “site” defined as a combination of four 
entities: 


1. a CSR-funded school, 


2. ademographically matched non-CSR school that did not receive any federal CSR funds 
before 2002 and is located in the same district as the CSR-funded school, 


3. the district within which the two schools are located, and 
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4. the state education agency in the state within which the district is located. 


The first phase of case study data collection occurred from spring 2003 to spring 2004. A second 
phase of site visits occurred during the 2004—05 school year. During visits to the 15 pairs of 
schools, the evaluation team used a formal observation instrument to observe classrooms ; 
reviewed school documents and materials; and interviewed state and district officials, school 
administrators, teachers, reform developers, and parents. 


A new round of case studies was conducted in spring 2007 through fall 2007 to extend the 
analyses of the earlier case studies. This study, known as the Improving Schools Study, involved 
13 schools identified by WestEd. Of this group of 13 schools, 9 were considered “improving 
schools,” and the other 4 were “comparison schools.” Ten of the schools in this sample were new 
to the case study analysis, while 3 were from the original group of 30 case study schools. 
Furthermore, 7 of the improving schools were CSR schools, and 2 were non-CSR schools. One 
of the comparison schools was a CSR school, and 3 were not. Site visitors used the same data 
collection protocols that were used in the 2004—05 round of site visits to the original 15 case 
study sites. 


The case studies include information related to reform implementation as well as its relationship 
to district and state policies, including those policies that do not focus directly on comprehensive 
school reform. The case studies follow a formal protocol that calls for integrating information 

from documents, interviews, focus groups, observations, and quantitative data (see Appendix B). 


STUDY LIMITATIONS 


The multimethod approach to this evaluation, like most studies of practice (Lipsey, 2003), has 
limitations. 


All achievement analyses presented in this report rely on standardized school-level achievement 
scores from NLSLSASD, CCSSO, EDEN and state Web sites. The use of school-level outcome 
measures presents methodological and psychometric challenges. School-level scores are 
aggregate measures of individual student performance within a school and, as such, do not 
account for variation in academic performance among students within a group. In addition, such 
analyses are not sensitive to the fact that performance may be contingent on both individual 
factors and factors related to group membership such as membership in different classrooms and 
school environments. The multileveled structure of data is not adequately addressed in aggregate 
analyses, which limits the precision of the estimates produced. Three major areas have been 
widely acknowledged as contributing to imprecision in aggregate analyses: aggregation bias, 
misestimated standard errors, and heterogeneity of regression (Burstein, 1978; Burstein & Miller, 
1981; Haney, 1980; Raudenbush & Bryk, 2002): 


1. Aggregation bias exists because variables at different organizational levels have different 
contextual meanings and may show varying relationships across organizational levels. 
Analyses based on aggregate outcomes do not account for differences in variables across 
organizational levels, leading to biased estimates of effects. Raudenbush and Bryk (2002) 
use socioeconomic status to explicate aggregation bias. They note that both individual 
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social class and socioeconomic status of the school influence student achievement. 
School-level socioeconomic measures may influence achievement beyond the effects 
estimated at the individual level. 


2. Misestimation of standard errors occurs in single-level analyses because such analyses 
ignore the similarities based on common experiences among responses from individuals 
within a group. For example, all students in a classroom are likely to be exposed to the 
same curriculum. Multilevel analyses calculate standard errors in a way that accounts for 
the clustering of individuals and adjusts for the dependence of responses from a particular 


group. 


3. Heterogeneity of regression is an issue because, despite the common experiences, 
differences exist within all groups. Consequently, the relationships between individual 
characteristics and outcomes may vary among individuals. This heterogeneity is masked 
in school-level analyses that assume a single linear relationship between characteristics 
and outcomes for all individuals within a group. In contrast, multilevel models provide a 
mechanism to explore the heterogeneity of relationships and potential moderating factors 
that may account for differences in the relationships between outcomes and explanatory 
variables. For example, the amount of school resources or level of teacher professional 
development in schools may influence the relationship between CSR and student 
achievement. 


Despite these weaknesses, designating the school as the unit of analysis and using aggregate 
outcome measures is deemed appropriate given the goals of the evaluation and the nature of the 
CSR program. The analyses rely on aggregate outcome measures that are both meaningful to 
schools and relevant to the current policy environment. School-level proficiency is the 
performance measure for which all schools are held accountable. Although the use of aggregate 
measures may compromise precision, its relevance for school officials and policy-makers offsets 
its limitations. Moreover, the CSR program was designed to stimulate comprehensive change 
through a set of coordinated reform actions that influence school operations across the whole 
school. Thus, it is appropriate to use the school as the unit of analysis and school-level 
achievement measures to examine how the CSR program influences achievement. 


Furthermore, although a multilevel model, in which students are nested within schools, would 
provide more precise estimates of the relationships between CSR participation and achievement 
outcomes than the models used in this evaluation, collecting individual student records over an 
extended time period from a large nationally representative sample of CSR and comparison 
schools places a heavy burden on schools. 


A second limitation of this study is shared with other efforts to assess program effectiveness 
across states over time. Such evaluations must overcome the difficulty of using the existing state 
assessments, which are designed to provide information about students’ progress toward 
mastering the content established in each state’s standards. Consequently, assessments differ. In 
fact, neither the content nor the criteria for determining proficiency are the same from state to 
state. Also, standards, assessments, and proficiency criteria often change, making scores within 
states difficult to compare over time. Finally, because of the structure of state assessments 
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(especially before NCLB), we were unable to track cohorts of students because such data were 
rare before 2004—05. School-level assessment scores were standardized within states for each 
year to analyze outcomes across states and assessment instruments.!8 The standardization 
provides a common metric for all achievement outcomes; however, it does not account for all 
sources of heterogeneity among states resulting from differences in assessments, the stringency 
of state proficiency standards, and the fact that the analyses rely on different groups of students 
in each year. A more precise analytic approach would use a multilevel model of schools nested 
within states to account for the variation across states. Insufficient within-state samples, 
stemming from both missing data and the manner in which 2002 CSR grants were awarded 
across states, precluded the use of a multilevel model to explore the relationships between CSR 
participation and student academic achievement. 


Finally, this study may also be limited by nonrandom attrition of a large number of schools from 
some analyses because of survey nonresponse, the fact that several states did not report all data 
to the Common Core of Data, and the removal of non-CSR schools that received CSR awards 
after 2002. The unrepresentative nature of missing data in the Year 5 sample, and the 
inadvisability of imputing such data from Year 3 responses, were factors precluding analyses in 
this report on the relationship between implementation and achievement. Thus, the failure to find 
a program effect of CSR participation may be a result of limitations in the design of this study as 
well as the lack of a relationship between the CSR program, its implementation, and achievement 
outcomes. 


18. See Exhibit 2 on pages 9-16 in the Third Year Report (U.S. Department of Education, 2008) for a list of 
the measures used for each state for the years 1999-2000 to 2004—05 of this study. This report updates those 
data with percent proficient in 2006-07; they were the only outcome measures available. 
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Ill. OVERALL RELATIONSHIP BETWEEN CSR AWARD AND 
ACHIEVEMENT 


Key Finding 


Was receipt of a CSR award associated with improvements in school-level 
mathematics and reading achievement? 


> Five years after initially receiving their CSR awards, schools 
receiving awards did not demonstrate larger achievement growth than 
matched comparison schools not receiving CSR grants. In fact, 
achievement gains were nonexistent in CSR elementary schools and 
were lower than their comparison schools at the middle school level. 


METHODOLOGY USED TO ASSESS THE RELATIONSHIP BETWEEN RECEIPT OF ACSR 
AWARD AND ACHIEVEMENT 


Analyses of the relationship between CSR award and growth in achievement were restricted to 
Title I schools that received CSR program funding. The evaluation team selected Title I schools 
using demographic information from the Common Core of Data, the U.S. Department of 
Education’s national database of school demographics, and the standardized achievement scores 
developed for this evaluation. Selection of the comparison group was based on data from 2001-— 
02, the year before the CSR schools began implementing their awards. 


States targeted CSR awards to low-performing Title I schools, resulting in 96 percent of CSR 
awards going to such schools. Therefore, analyzing Title I schools allows for inference regarding 
the population of greatest interest. However, Title I encompasses a wide range of schools; in 
2003-04, more than 54,000 schools received Title I assistance as either a Schoolwide Title I or 
Targeted Assistance School (U.S. Department of Education, 2007). Given the emphasis most 
states placed on making CSR awards to the lowest performing and highest poverty schools, it 
comes as no surprise that the population of all Title I non-CSR schools is higher performing, is 
smaller, and contains lower percentages of students from traditionally underserved minority 
groups and those eligible for free and reduced-price lunch than Title I CSR schools (Exhibit 5). 
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Exhibit 5 
Differences Between the 2002 Cohort of Title | CSR Schools and Title | Non-CSR 
Schools in Achievement, School Size, and Demographics in 2001-02 


Non-CSR 


CSR 


Average 


N 


Average 


Difference 


Standardized Assessment Scores 


Elementary Mathematics (SD) 


-0.20 


273 


-0.87 


0.67** 


Elementary Reading (SD) 
Middle School Mathematics (SD) 


-0.22 
-0.16 


274 
105 


-0.90 
-0.94 


0.68** 
0.78** 


Middle School Reading (SD) 


-0.17 


105 


-0.95 


0.78** 


Demographic and Scho 


ol Characteristics 


Membership (N) 


479 


402 


544 


-65** 


Minority (%) 
FRL (%) 


48,872 


35.2 
51.0 


386 
362 


54.5 
70.5 


-19.3** 
-19.5** 


Exhibit highlights: Achievement in Title I Comprehensive School Reform (CSR) schools in the 
2001-02 school year was approximately three-fourths of a standard deviation lower than in 
Title I non-CSR schools. The percentage of minority students in Title I CSR schools was about 
19 percent higher, and the percentage of students eligible for free and reduced-price lunch was 
about 21 percent higher than in Title I non-CSR schools. 


Source: Common Core of Data (CCD). 


Note: ** p < .01. “FRL” refers to students eligible for free or reduced-price lunches. All data 
are for the 2001—02 school year to demonstrate baseline achievement and demographics. These 
data are restricted to those schools for which achievement data were available in both 2002 and 
2007 (see Exhibit 2 for a list of these states). Standard errors are included in Exhibit A.1. 


Because of this disparity, the evaluation team developed a comparison group of non-CSR 
schools for the analysis of the relationship between receiving a CSR award and achievement, 
using Mahalanobis propensity scoring (Rosenbaum & Rubin, 1983, 1985; Rubin, 1980) based on 
mathematics and reading achievement, percentage of free and reduced-price lunch, and 
percentage of minority.!9 In this instance, CSR schools are counted as treated schools and non- 
CSR schools are counted as untreated schools. For each CSR school, Mahalanobis propensity 
scoring chooses a comparison school from the pool of untreated schools that most closely 
matches the CSR school based on the achievement and demographic measures used.?° 


Because some states did not report percentage of free and reduced-price lunch, and because 
sometimes CSR schools were located in small districts, the propensity score matching was run 
several times, first starting with the most restrictive matching procedure and then gradually 


19. Note that the comparison group developed for the analysis of the relationship between receiving an award 
and achievement gains is different from the comparison group developed at the outset of this study. The 
comparison group used in this section is based only on Title I schools; furthermore, it relies on updated CSR 
award, Common Core of Data, and NLSLSASD data files not available at the outset of this study. 

20. The Mahalanobis propensity score matching was completed using the -mahapick- module in Stata (Kantor, 
2006). 
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relaxing the matching conditions. All matches are based on, at the minimum, mathematics and 
reading achievement and percentage of minority. Matches were restricted to being located in the 
same state and having the same school configuration (elementary, K—8, or 6—8 middle school) as 
the CSR school. The conditions that were used are listed from most restrictive to least restrictive. 
The comparison school was chosen from the most restrictive set of conditions where a successful 
comparison school was found. 


1. The first set of matches is based on percentage of free and reduced-price lunch and being 
located in the same school district. Comparison schools selected in this group most 
closely matched their CSR school on mathematics and reading achievement, percentage 
of minority, and percentage of free and reduced-price lunch; were located in the same 
school district as the CSR school; and had the same school configuration as the CSR 
school. 


2. The second set of matches removes free and reduced-price lunch from the restrictions 
used for the first set. 


3. The third set of matches is similar to the first, but instead of restricting the matching to 
the same district, restricts the matching to the same type of locality (e.g., large city, small 
city, suburban, or rural) in the state. This approach expands the pool of potential matches 
for schools in small school districts that may not have many potential matches. 


4. The fourth set of matches removes free and reduced-price lunch from the third set of 
matches. 


5. The fifth set of matches includes free and reduced-price lunch and removes any locality 
restriction (choosing any school in the state). 


6. The last set of matches removes free and reduced-price lunch from the restrictions used 
for the fifth set of matches. 


The next two exhibits provide data for CSR schools and their comparison schools selected by the 
propensity-score matching process described above.*! Exhibit 6 presents average school-level 
achievement scores and demographics for schools with elementary mathematics and reading 
scores, while Exhibit 7 presents these measures for schools with middle school mathematics and 
reading scores. The CSR and comparison schools are very closely matched on baseline 
achievement and demographics. 


21. Note that this group of schools chosen for the achievement analysis is different from the group of schools 
chosen at the outset of the study to document implementation (see pages 12—16). The group of schools chosen 
for the achievement analysis encompasses all Title I CSR schools and matched comparison Title I schools; the 
selection process was done toward the end of this study and is based on the most recent data available for CSR 
schools, which had been updated as recently as spring 2007. The group of CSR schools and their matched 
comparison schools chosen at the outset of this study to document implementation is based on data that were 
available only at the beginning of this study; furthermore, it is a sample drawn from the population of CSR 
schools. 
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Exhibit 6 


Differences Between the 2002 Cohort of Title | CSR Schools and Non-CSR Title | 
Schools Chosen by Propensity Scoring: Schools With Elementary Mathematics and 


Reading Achievement 


Non-CSR 


CSR 


Average 


N 


Average 


Difference 


Standardized Assessment Scores 


Elementary Mathematics 


-0.79 


234 


-0.77 


-0.02 


Elementary Reading 


-0.77 


235 


-0.81 


0.04 


Demographic and Scho 


ol Characteristics 


Membership (N) 


478 


530 


244 


497 


Minority (%) 


478 


56.2 


244 


53.1 


FRL (%) 


461 


68.8 


233 


71.1 


Exhibit highlights: There are no statistically significant differences between matched 
Comprehensive School Reform (CSR) and non-CSR Title I schools in the baseline (2001-02) 
school year. 


Source: CSR Awards Database; Common Core of Data (CCD); National Longitudinal School- 
Level State Assessment Score Database (NLSLSASD). 


Note: “FRL” refers to students eligible for free or reduced-price lunches. These data are 
restricted to those schools for which achievement data were available in both 2002 and 2007 


(see Exhibit 2 for a list of these states). Standard errors are included in Exhibit A.2. 
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Exhibit 7 
Differences Between the 2002 Cohort of Title | CSR Schools and Non-CSR Title | 
Schools Chosen by Propensity Scoring: Schools with Middle School Mathematics and 
Reading Achievement 


Non-CSR CSR 


N Average N Average Difference 


Standardized Assessment Scores 


Middle School Mathematics 248 -0.66 95 -0.74 0.08 


Middle School Reading 248 -0.63 95 -0.63 0.06 


Demographic and School Characteristics 
Membership (N 248 675 109 615 
Minority(% 248 55.9 109 51.1 
FRL (% 243 65.1 103 65.3 


Exhibit highlights: There are no statistically significant differences between matched 
Comprehensive School Reform (CSR) and non-CSR Title I schools in the baseline (2001-02) 
school year. 


Source: CSR Awards Database; Common Core of Data (CCD); National Longitudinal School- 
Level State Assessment Score Database (NLSLSASD). 


Note: “FRL” refers to students eligible for free or reduced-price lunches. These data are 
restricted to those schools for which achievement data were available in both 2002 and 2007 
(see Exhibit 2 for a list of these states). Standard errors are included in Exhibit A.3. 


After the selection of the comparison schools, paired t-tests were used to assess whether the 
change in achievement from 2002—03 to 2004—05 is different between CSR schools and their 
comparisons (Zhao, Li, Gao, & Tierney, 2001; Fraas, Newman, & Pool, 2007).”2 


FINDINGS 


Was receipt of a CSR award associated with improvements in school-level mathematics 
and reading achievement? 


>» Five years after initially receiving their CSR awards, schools receiving awards did 
not demonstrate larger achievement growth than matched comparison schools not 
receiving CSR grants. In fact, achievement gains were nonexistent in CSR 
elementary schools, were marginally lower than their comparison schools in 
middle school mathematics, and were no different from their comparison schools 
in middle school reading. 


At the elementary school level, changes in mathematics and reading achievement are not 
statistically significant for both the CSR and matched non-CSR schools (Exhibit 8). At the 


22. T-tests are used to test the null hypothesis that the means of two groups (in this instance, CSR and 
comparison schools) are the same. A /-statistic is calculated from the two groups’ means and standard 
deviations, which is then used to determine the probability of correctly rejecting the null hypothesis of no 
relationship by comparing it with students’ ¢-distribution. 
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middle school level, changes in mathematics achievement are statistically significant for CSR 
schools but not for non-CSR schools. Differences in achievement change between CSR and non- 
CSR schools are statistically significant for mathematics achievement, although only at the .10 
level, while differences in changes in reading achievement are statistically insignificant. Thus, 
after five years, schools receiving three-year CSR grants in 2002 were no better at improving 
achievement than Title I schools that were similar in demographics and achievement in the 
baseline year. 


Exhibit 8 
Changes in Standardized Assessment Scores in CSR and Matched Non-CSR Schools 
from 2002-03 to 2006-07 


Changes in Standardized Assessment Scores 
(SD) 


Non-CSR 


N 


CSR Schools 


Schools 


Difference 


Elementary Mathematics 


493 


0.04 


0.00 


0.04 


Elementary Reading 


563 


0.02 


0.04 


-0.02 


Middle School Mathematics 


291 


0.01* 


0.11** 


-0.10+ 


Middle School Reading 


294 


0.04 


0.10* 


-0.06 


Exhibit highlights: Community School Reform (CSR) schools had statistically significant 
increases between 2002-03 and 2006-07 in middle school mathematics of 0.01 standard 
deviations. However, matched non-CSR schools had larger statistically significant increases in 
middle school mathematics and reading of 0.11 and 0.10 standard deviations, respectively, 
although only middle school mathematics had a marginally statistically significant difference 
between CSR schools and their matched non-CSR schools. 


Source: CSR Awards Database; Common Core of Data (CCD); National Longitudinal School- 
Level State Assessment Score Database (NLSLSASD); Education Data Exchange Network 
(EDEN); Council of Chief State School Officers (CCSSO); and state Web sites. 


Note: + p < .10; * p< .05; ** p< .01. Tests for the statistical significance of achievement gains for 
CSR and non-CSR schools are t-tests to assess whether the value is different from zero. The 
differences between CSR and non-CSR schools are assessed through paired ¢-tests. Standard errors 
are included in Exhibit A.4. 


DISCUSSION 


Unlike these results, earlier CSR research shows an increase in mathematics and reading 
achievement over multiple years of implementation (Borman, Hewes, & Overman, 2003). Two 
possible reasons for the lack of findings here are suggested by the initial case study of 15 pairs of 
schools. First, a number of schools ended implementation of an original CSR strategy, replacing 
it with a portfolio of programs. Second, some schools added reform programs, creating 
competition for teachers’ time for professional development. 


School W, for example, chose the Comer School Development model after award receipt in 
2002. However, a new principal arrived in 2003 and instead chose the Effective Schools model. 
Teachers indicated they were not involved in this decision and had already scheduled 
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professional development for the original model for the summer. Staff members then engaged in 
what they termed a “whirlwind” of professional development on the Effective Schools model. 
However, key consultants for the Effective Schools model became ill, compounding staff 
indifference to the program. Ultimately, neither the Effective Schools model nor the Comer 
School Development model was implemented. 


Another school, School E, was encouraged by the district’s then superintendent to adopt and 
implement Accelerated Schools. The external assistance provided by the model developer helped 
the school arrange common planning time for teachers and offered more than 20 days of 
professional development in 2002—03. However, in 2003, district leadership changed, and the 
new superintendent focused efforts on supplementary programs. Teachers were required to 
attend professional development sessions for the district’s priority programs, creating scheduling 
conflicts. By 2004, Accelerated Schools was no longer present in the school. 


Thus, implementation (or lack thereof) may explain the overall lack of a relationship between 
receiving a CSR award and achievement gains. The next section provides data results of the 
survey as well as case studies on the implementation of the 11 components and reports on the 
comprehensiveness of CSR implementation, as measured by the number of components a school 
implements. 
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IV. THE COMPREHENSIVENESS OF CSR IMPLEMENTATION 


Key Finding 


Were schools that received CSR awards more likely to implement the 
legislatively specified components of CSR than other schools? 


> No, consistent with earlier study findings, both CSR and non-CSR 


schools implemented a similar number of legislative components in 
both 2003 and 2007. The number of components rose slightly for 
CSR and non-CSR schools during this period at both the elementary 
and middle school levels. 


This report includes analyses of changes in the extent to which both CSR and non-CSR schools 
implemented components of the CSR program over a five-year period. This chapter focuses on 
the implementation of CSR components, comparing schools receiving three-year CSR grants in 
2002 with comparison schools that did not receive such grants. It also includes the findings from 
the case studies of CSR and non-CSR schools. 


MEASURING IMPLEMENTATION OF THE CSR COMPONENTS 


The analyses of the CSR components rely on measures of the CSR components developed from 
the surveys administered in 2003, 2005 and 2007 to both CSR and non-CSR schools. The 
surveys ask principals and teachers about the implementation of activities consistent with the 11 
components identified in NCLB. For most questions, respondents were presented with a forced 
choice of “0” for not having implemented the activity and “1” for implementing the activity.?3 
Within each component, respondents’ answers were averaged to develop a component score for 
each school. 


The component scores were converted into measures of whether a school is counted as 
implementing a component. That is, schools were assigned a 0 if their component score or their 
responses to survey questions fell below a certain threshold and a | if they exceeded the 


23. This evaluation measures implementation differently from Aladjem et al. (2006), who measured 
implementation as the difference between what the model developers consider to be full implementation and 
what the school actually does. To do so, they surveyed each of the model developers and asked them to 
respond as if they were a school that implemented their model. The researchers compared each school’s 
responses to this measure of implementation to develop a measure of fidelity. To measure implementation in 
the comparison schools, Aladjem et al. predicted what CSR model the comparison school would have chosen 
based on its school characteristics and those of the CSR schools. The researchers then compared the school’s 
responses with those of the model developer of the predicted CSR model. This measure of implementation is 
consistent with the focus on models as compared with this study’s focus on fidelity with the 11 components 
identified in NCLB. 
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threshold. For most of the components, the school had to report that it was implementing all of 
the activities to be counted as implementing the component. In some instances, however, this 
requirement would have resulted in no schools implementing the component, so the threshold 
was relaxed somewhat. Also, for parental involvement, the measures used were not dichotomies 
but, instead, were estimates of the percentage of parents engaging in various activities in the 
school; another threshold was developed based on the distribution of the estimates. The metric 
used to measure whether a school implemented a component is included in the discussions of 
how each component is measured. This conversion process was done to measure whether a 
school implemented a component and was used to calculate the number of components 
implemented. 


The surveys were modified between the three administrations as a result of findings from the 
2003 survey concerning the existence of multiple reforms in schools, the need to develop more 
refined measures of professional development, and the fact that some of the questions in the 
2003 survey were no longer applicable in 2005 and 2007. For instance, in the 2003 survey, 
schools often reported implementing more than one reform method. The 2005 survey was 
modified to capture information on up to four reforms the school was undertaking (see Appendix 
B). As a result of these changes, some of the components were not measured consistently across 
the two time periods; these modifications also required changes in assessing whether a 
component was implemented. All such changes are documented below in the discussion of each 
component. As a result, comparisons on some components across time were limited or 
unavailable. 


In addition to the survey results, the results of the case studies are integrated in the sections on 
components below. Two different sets of case studies were conducted during this longitudinal 
study. The initial case studies consisted of 15 pairs of schools from 15 districts across 14 states— 
with each pair containing a CSR school and a non-CSR school—and were selected to 
supplement the quantitative analyses with qualitative understanding of implementation 
dynamics. With such a small number of case study schools, this sample does not represent either 
the geographic distribution or the distribution of school levels in the universe of CSR schools. 
However, the selection ensured that the case study sample includes enough geographic and 
school-level diversity to reflect a variety of state policy environments, including schools in states 
with a long tradition of local control and others in states that exercise more centralized control 
over schools. Site visitors paid three visits to these schools, in 2003, 2005 and 2007. 


The COSMOS Corporation conducted an additional round of data collection beyond the original 
30 case study schools. This round of case studies, known as the Improving Schools Study, 
involved 13 schools identified by WestEd. Of this group, 9 were considered “improving 
schools,” and the other 4 were “comparison schools.” Ten of the schools in this sample were new 
to the case study analysis, while 3 were from the original group of 30 case study schools. 
Furthermore, 7 of the improving schools were CSR schools, and 2 were non-CSR schools. One 
of the comparison schools was a CSR school, and 3 were not. The 13 schools in the Improving 
Schools Study were visited in spring through fall 2007. Site visitors relied on the same site visit 
protocols as were used in the original 15 case study sites during the 200405 site visits. 
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Components 1 and 11—Research-Based Design and Evidence-Based Practice 


Given the substantial similarity between Research-Based Design and Evidence-Based Practice 
components, the decision was made to combine them into one measure. As defined in NCLB, 
evidence-based practice focuses on proven methods and strategies for student learning, teaching 
and school management that reflect scientifically based research and effective practices 
replicated successfully in schools with diverse characteristics. Added in NCLB, the research- 
based design component calls for the adoption of programs that use scientifically based research 
to document either significant improvement in the academic achievement of students 
participating in such programs as compared with students in schools that have not participated in 
such programs or the existence of strong evidence that such programs will significantly improve 
the academic achievement of participating children. 


The 2003 measure consists of a principal’s responses to three questions, each of which was a yes 
or no question on the survey: whether there was evidence based on independent research 
supporting the model (33 percent indicating yes), whether there was evidence based on research 
conducted by the reform developer (34 percent indicating yes), and whether there was evidence 
that relied on the use of control or comparison groups (21 percent indicating yes). The 2003 
measure was calculated as the average of these three items (Exhibit 9). A school was counted as 
implementing this component if it reported that at least two of the three types of evidence were 
present. 


The 2005 measure consists of similar items as the 2003 measure: whether the reform model has 
evidence that it improves student achievement (89 percent indicating yes), relies on a comparison 
or control group (60 percent indicating yes), and was shown to improve student achievement at 
other schools (91 percent indicating yes). Each of these items was also a yes or no question. 
Nonmissing responses were totaled, and the school was counted as implementing this component 
if the average of the nonmissing responses was |. Thus, if a school reported on only two of the 
measures and reported that it implemented both of them, then that school was counted as 
implementing a research-based design. 


The 2007 measure relies on principals’ identification of three sources of evidence that contribute 
to the selection of reform. Fifty-nine percent of principals indicated that the theoretical or 
research foundation of the reform model was important in its selection; 55 percent stated that 
successful implementation at other schools was important; and 21 percent indicated that the best 
published results were important. The measure for implementation was constructed in the same 
manner as in previous years. 
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Exhibit 9 
Means and Standard Deviations of Survey Items Used to Construct Research-Based 
Design and Evidence-Based Practice Measure 


| __evidencettems |_| Moan | Deviation | 

Evidence Items Deviation 
Independent research 
Research by reform designer 
Comparison or control groups 
Improves achievement 
Comparison or control groups | 632 | o60 | 045 | 
Student achievement at other schools 

2007 | Theoretical or research foundation 
Successful implementation at other schools 
Best published results 


Exhibit highlights: The survey items used to construct the research-based design and evidence- 
based practice measure changed from year to year. The reader may compute the means into a 
percentage by multiplying the mean value by 100. For example, the 2003 mean score of 0.21 for 
comparison or control groups indicates that 21 percent of respondents indicated that the reform 
method they chose had evidence based on comparison or control groups. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) Surveys. 


Case Study Findings 


Nearly all of the 15 matched-pair original case study schools reported the use of scientifically 
based research. In three instances, there was no information or evidence of use in CSR schools, 
and in another CSR school, the reform effort had been dropped. Two of the comparison schools 
were reported to be having “issues with implementation” of scientifically based research to 
support reform efforts. 


Component 2: Comprehensive Planning 


Questions on the survey forms measure two aspects of comprehensive planning—classroom- 
based planning and school-based planning. Surveys asked respondents about both aspects 
consistently in all survey years. On all questions, respondents were asked whether their school 
improvement plans contained nine components of planning; they could answer yes or no to each. 
The classroom-based measures include curriculum and instruction, student assessment, 
classroom management, and professional development. Results show that there was an increase 
of schools identifying the items in their school improvement plans between 2003 and 2005, 
followed by a decrease in the number of schools in each item in 2007 (Exhibit 10). The school- 
based planning measure includes measurable goals for reform, periodic evaluation, parental 
involvement, professional development, participation in school management, and integration of 
new technology. As with the previous exhibit, the same trend of reporting appears with an 
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increase in items between 2003 and 2005 followed by a decrease in 2007 (Exhibit 11). Note that 
professional development occurs in both school- and classroom-level planning because 
professional development opportunities may be around subject-specific topics or school-reform 
related topics. In all years, a school was counted as implementing these components if it reported 
that the school improvement plan contained all of the items. 


Exhibit 10 
Means and Standard Deviations of Survey Items Used to Construct Comprehensive 
Planning-Classroom Measure, an Year 


Aspects of Reform | = 2007,——sd*F 
Covered by School Standard Se Standard 
Improvement Plan Deviation Deviation eri 
Curriculum and instruction . | 0.39 39 


| 


Exhibit highlights: The survey items used to construct the comprehensive planning-classroom 
measure were consistent across years. The reader may compute the means into a percentage by 
multiplying the mean value by 100. For example, the 2003 mean curriculum and instruction 
score of 0.88 indicates that 88 percent of respondents indicated that the school improvement 
plan covered curriculum and instruction. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) Surveys. 


Note: There are 720 observations in 2003, 673 observations in 2005, and 536 observations in 
2007. 
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Exhibit 11 
Means and Standard Deviations of Survey Items Used to Construct Comprehensive 
Planning-School Measure, by Year 


Aspects of Reform | 2005” 2007 

Covered by School Standard Standard Standard 
Improvement Plan Deviation Deviation Deviation 
Measurable goals : 
Periodic evaluation 


Participation in school 0.63 0.49 


management 


New technology 


Exhibit highlights: The survey items used to construct the comprehensive planning-school 
measure were consistent across years. The reader may compute the means into a percentage by 
multiplying the mean value by 100. For example, the 2003 mean parental involvement score of 
0.80 indicates that 80 percent of respondents indicated that their school improvement plan 
covered parental involvement. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) Surveys. 


Note: There are 720 observations in 2003, 673 observations in 2005, and 536 observations in 
2007. 


Case Study Findings 


In spite of the fact that coherent strategies imply planned and systematic approaches, there was 
little documented evidence in any of the original 15 case study sites that such comprehensive 
planning was used to address the reform strategies. Planning was primarily focused on 
instruction and use of assessment data. While data were widely used, it was not clear how these 
data were used to implement or modify the CSR program or other initiatives. Some schools 
found what formative evaluations were done to be helpful, but no explanations were offered as to 
how these evaluations resulted in plan modifications or adjustments. 


The 13 improving schools case study sites used complementary research-based supporting 
strategies that seemed to indicate a well-planned approach to reform. For example, nearly half of 
the schools used differentiated learning as a supporting strategy for the primary approach to 
literacy instruction. In another example, 9 of 13 schools implemented whole-school literacy 
reform across all grades, suggesting a coherent approach. 


Component 3: Professional Development 


The professional development component relies on the reported number of days of professional 
development; however, it was measured differently across survey years and any changes should 
be interpreted cautiously. On the 2003 survey, respondents were asked whether their school 
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provided at least 10 days of professional development. Principals could respond yes or no, and 
the school was counted as fully implementing the measure if they responded yes. On the 2005 
and 2007 surveys, teachers were asked to report the number of hours of professional 
development they received in the last year. For analysis and reporting, the teacher’s response was 
converted to days by taking the number of hours and dividing by six (approximately the length of 
an average school day). Schools were counted as implementing professional development if the 
average number of days of professional development received was 10 or more. The result of the 
survey shows that in 2005, 65 percent of schools provided 10 or more days of professional 
development as reported by teachers, which was an increase from 2003 results provided by the 
principals (48 percent). There was a decrease in the number of schools that provided 10 or more 
professional development days, as reported by teachers, in 2007 compared with 2005 (Exhibit 
12). 


Exhibit 12 
Means and Standard Deviations of the Survey Item Used to Construct the Professional 
Development Measure, cerns Year 


| = 2007,— —=* 


a 
Development Standard Standard Standard 
Item cere lit er ee 


Professional 
Development } 605 | 048 | 0.48 : 714 | | 048 | 48 0.50 | 50 


Exhibit highlights: The reader may compute the means into a percentage = ac the 
mean value by 100. For example, the 2003 mean professional development score of 0.48 
indicates that 48 percent of respondents indicated that the school offered at least 10 days of 
professional development to teachers. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) Surveys. 


Case Study Findings 


Within the case study schools, two different findings emerged. First, while documentation of 
professional development hours did not occur for all case study schools, professional 
development activities seemed to be more sustained at some schools. At one school for example, 
a database was used to document teacher professional development activities that were provided 
to each staff member and how these activities were linked to both the school and individual 
professional development plans. Stipends were available to teachers who exceeded 24 hours of 
professional development. Teachers averaged between 24 and 34 hours per year in Year 2 of the 
study and 20 to 40 hours in Year 3. 


Second, all but 2 of the 13 schools in the improving schools case study used literacy coaches to 
support teachers in implementing specific strategies. Numbers of coaches per school varied from 
one or two who focused on both mathematics and literacy to one high-performing school with 
one coach and five reading specialists. Coaches not only provided professional development but 
also assisted teachers one-on-one, modeled lessons within classrooms, and often led grade-level 
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and vertical planning sessions. Coaches were also responsible for keeping track of student 
performance. One school reported providing more than 1,700 hours of coaching support to 
teachers throughout the school year. 


Component 4: Goals and Benchmarks 


The goals and benchmark measures rely on a somewhat different set of items across survey years 
and any change should be interpreted cautiously. Four questions from the 2003 surveys were 
used to construct the measure for goals and benchmarks (see Exhibit 13): whether the school 
improvement plan includes student assessment rubrics (50 percent indicating yes), whether the 
school improvement plan includes measurable goals and objectives (97 percent), whether the 
school has end-of-year student achievement goals (88 percent), and whether the school has 
interim student achievement goals (70 percent). Nonmissing responses were totaled, and the 
school was counted as implementing this component if the average of the nonmissing responses 
was |. Thus, if a school reported on two of the measures and indicated those were in place, the 
school was counted as implementing goals and benchmarks. 


The 2005 goals and benchmark measure relies on the same two school improvement plan items 
as the 2003 measure, that is, student assessment rubric (64 percent) and measurable goals and 
objectives (98 percent). However, the two questions about student goals were not asked in the 
2005 surveys. Instead, respondents were asked whether the reform strategies were accompanied 
by implementation benchmarks (75 percent indicating yes). One of the earlier findings in this 
study is that schools were often implementing more than one reform strategy simultaneously. As 
a result, reform-specific questions were asked four times for each respondent. In some cases, the 
respondent indicated that there was only one reform going on in the school; in other cases, 
respondents listed up to four reform strategies. The responses were averaged to derive a school- 
level measure for this item that ranged between 0 and 1. The 2005 goals and benchmarks 
measure was created by averaging the nonmissing responses; schools with a score of 1 were 
counted as implementing goals and benchmarks. 


The 2007 goals and benchmark measure relies on the same two school improvement plan items 
as the measures from previous years, that is, student assessment rubric (36 percent) and 
measurable goals and objectives (95 percent). In addition, the 2007 measure includes an item on 
whether the school improvement plan included a mechanism for periodic evaluation of goals (80 
percent indicating yes). The 2007 goals and benchmarks measure was created by averaging the 
nonmissing responses; schools with a score of 1 were counted as implementing goals and 
benchmarks. 
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Exhibit 13 
Means and Standard Deviations of Survey Items Used to Construct the Goals and 


Benchmarks Measure, by Year 
Standard 
Year Goals and Benchmark Items Deviation 


2003 | Does the SIP include student assessment rubrics? 
Does the SIP include measurable goals or objectives? 


Does the school have student goals at the end of the 
year? : 


Does the school have intermediate student goals? 


Exhibit highlights: The goals and benchmarks measures rely on a somewhat different set of 
items across survey years and any change should be interpreted cautiously. The reader may 
compute the means into a percentage by multiplying the mean value by 100. For example, the 
2003 mean score of 0.50 for whether the school improvement plan includes student assessment 
rubrics indicates that 50 percent of respondents indicated that the school improvement plan 
includes student assessment rubrics. 


Does the SIP include measurable goals or objectives? 


Strategies accompanied by implementation 
benchmarks? 

2007 | Does the SIP include student assessment rubrics? 
Does the SIP include measurable goals or objectives? 


Does the SIP include periodic evaluation of goals? 


| 
Does the SIP include student assessment rubrics? 
3 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) Surveys. 


Note: “SIP” refers to school improvement plan. 


Case Study Findings 


The use of data-driven decision making was evident in case study schools. However, in almost 
all cases, use of data for the specific objective of benchmarking and goal setting was not clearly 
identified. Case study schools approached this task in a variety of ways, including test analysis 
by grade level, student performance across grades (involving vertical planning), use of data 
coaches, and use of technology to support and facilitate analysis. Emphasis on the use of data 
seemed to be a function of the degree of district focus and support. In instances where use of data 
was the strongest, usually part of weekly team or planning meetings, similar findings were noted 
for both CSR schools and their matched comparison schools. In one instance, use of data was 
cited as part of the “school culture” for a CSR school and its comparison school. 


Use of data for instructional decision making was evident in case study schools. Use of 
assessments other than those mandated by the state or district or use of formative assessment 
strategies was less evident. In two instances both CSR schools and their comparison schools 
were noted for using a wide array of assessments. For one pair of schools, these assessments 
included state results from criterion-referenced and norm-referenced tests as well as a phonics 


Chapter IV 33 


assessment, reading assessments for all grades, district assessment data for each nine-week 
period, and alternative assessments for special education students. Only one school was reported 
to have used student work samples as part of its ongoing analysis of data. 


Case study schools were more likely to use data to assess and modify instruction than to evaluate 
school initiatives and programs. In one example, a school used results from the Iowa Tests of 
Basic Skills not only to examine performance of subgroups of students that missed AYP targets 
but also to determine what specific professional development was needed to address the needs of 
these students. Another school combined results from a state-provided “gap analysis” that 
identified misalignment between students’ scores and state standards, benchmarks from previous 
school improvement plans, and information from the formative evaluation process to revise the 
comprehensive school improvement plan. 


In the case study sites, training on data use was more likely to be provided by the district than by 
the state. In one district, principals received monthly training on how to make data-driven 
decisions and were expected to share training with their schools’ staff members. The district also 
organized a course for teachers at a local college on using data to inform decisions. 


As noted in Component 3 results from the improving schools case study, coaches tended to be 
the data keepers and provided the faculty with performance results by student and by classroom 
that could then be used to inform instruction. Most coaches kept data books with detailed 
performance findings. At two schools, coaches prepared “data rooms” where teachers met and 
planned lesson content. At one of these schools, the library wall had a color-coded card for each 
student with the student’s performance on each assessment as well as the prescribed intervention. 
Coaches updated the wall after each assessment, and during one site visit, clusters of teachers 
were observed at the wall discussing overall classroom and individual student performance. 


Component 5: Staff Participation 


The 2003 and 2005 measures for staff participation rely on almost the same survey items; the 
2003 measure includes one additional item that was not asked on the 2005 survey (Exhibit 14). 
Both measures rely on yes or no questions about all teachers participating in reform and about 
factors that limit participation (i.e., funding and subject-specific reform). The 2003 measure 
includes estimation by the principal of the percent of teachers who participate in reform. The 
2007 measure relies on principal survey items that asked whether administrators and teachers 
participated in a vote to implement the current reform (43 percent and 45 percent, respectively), 
whether there were no limitations on teacher participation in the reform (63 percent), and 
whether the reform has support from staff members (56 percent). The staff participation was an 
aggregate measure, which was developed by averaging all the nonmissing responses within a 
school; schools with a score of 1 were counted as implementing. 
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Exhibit 14 
Means and Standard Deviations of Survey Items Used to Construct Staff Participation 
Measure, by Year 


[vear| ____statParicipationtioms |__| _mean_| Deviation | 
Year Staff Participation Items Deviation 
2003 | Percentage of teachers who participate in reform 
All teachers participate in reform (yes or no) 
Participation is not limited by subject-specific reform 
Participation is not limited because of funding 
All teachers participate in reform (yes or no) 
Participation is not limited by subject-specific reform | 674 | 090 | 030 | 
Participation is not limited because of funding 
2007 | Administrators voted for the current reforms 
Teachers voted for the current reforms 
No limits on teacher participation 
Suppor from staf 


Exhibit highlights: Survey items used to measure staff participation changed each year of the 
survey. The reader may compute the means into a percentage by multiplying the mean value by 
100. For example, the 2003 mean score of 0.80 for whether all teachers participate in reform 
indicates that 80 percent of respondents indicated that all teachers participate in reform. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) Surveys. 


Case Study Findings 


The case studies pointed to several examples of staff support for school reform. In one school, 
charts and posters on walls throughout the school indicated participation in, and support of, the 
reform method by both teachers and students. The charts and posters reflected all of the methods 
and strategies being used. At another school, the faculty was initially unanimously behind the 
reform method but then voted to discontinue it after two years. This faculty remained involved in 
implementation of school improvement strategies through faculty meetings, grade-level 
planning, and curriculum content meetings. Finally, there were instances where faculty support, 
or lack thereof, was dependent on the principal. For example, one school had strong support for 
the reform method, yet when the program ended and a new principal took over, the new principal 
emphasized an individual approach to teaching in which teachers used their own methods to 
prepare students for the state assessment. 


Teachers in the Improving Schools Study were generally supportive of the reforms. In the case of 
K-8 schools, teachers at the lower grades were more supportive than teachers at the upper 
grades. In 7 of the 13 schools in the Improving Schools Study, staff members selected the 
research-based strategy. In one case where the principal selected the strategy, it was no longer in 
use in 2006-07. As noted earlier, coaches played a major role in providing teacher support for 
specific reforms strategies. 
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Component 6: District Support 


For 2003 and 2005, the district support measure is a combination of the same six items 
pertaining to the types of district support: needs assessment, additional staff, selecting a reform 
model, writing grant proposals, professional development, and release time for teachers. Each of 
these items is formatted as a yes or no question. All six items saw an increase in district support 
between 2003 and 2005, except writing grant proposals, which remained relatively consistent 
(Exhibit 15). For 2007, the district support measure is a combination of five different items 
pertaining to types of district support: developing school improvement plan (71 percent), 
reviewing school improvement plan (70 percent), providing data for school improvement plan 
review (81 percent), providing training on school improvement plan development or review (76 
percent), and establishing school goals or benchmarks (63 percent). In each year, the nonmissing 
scores were averaged, and a school with an average score of 0.8 or higher was counted as 
implementing this measure. 
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Exhibit 15 
Means and Standard Deviations of Survey Items Used to Construct the District Support 
Measure, by Year 


[vear | ___oistict supporttoms | _n_|_ean | Devaton | 
Year District Support Items Deviation 
aus 
|Releasetimeforteachers = sts—i—s—sOY time for teachers 0.49 
[Needs assessment | t= | 4 | 050 
[Additional staff wt | ts | 50 
|Selectingareformmodel Tt =| 03 | 
-aicina gant poposais _—_}_sii_}_a86_|__os0_ 
Professional |Professionaldevelopment = ssid 0.39 
[Release time forteachers SS |7t_| 063 | 0.48 
2007"||DevelopingachasliinprvemenOlay= || ines 0.74. 2) 0a 
| Reviewingiachoolimprovermentiplay = == | eae. SE vom 2 “06: =] 


| Providing data for school improvement planreview — || 536 =| ~=—0.81 | 0.39 


Providing training on school improvement plan a ea ee 
development or review 0.76 0.43 
Establishing school goals or benchmarks | 536 | 063 | 043 | 


Exhibit highlights: Although survey items used to measure district support were the same in 
2003 and 2005, they changed in 2007. The reader may compute the means into a percentage by 
multiplying the mean value by 100. For example, the 2003 mean needs assessment score of 0.45 
indicates that 45 percent of respondents indicated that the district assisted the school in 
conducting a needs assessment. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) Surveys. 


Case Study Findings 


The most widespread strategy in case study schools was for the district to provide coaches to 
assist in schools. Peer observations and reviews were the most frequently mentioned use of 
coaches. Coaches were also reported to mentor new teachers in several schools while another 
school specifically noted lesson modeling provided by a coach. 


Several schools in the improving schools case study used bonuses as a teacher retention strategy. 
At one school, bonuses based on achievement of growth targets varied from $3,700 to $8,000 per 
teacher. Although the source of these funds was not mentioned, the amount suggests district 
support at the very least. At another school, teachers received extended pay for working in an 
inner-city school. 
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Component 7: Parental Involvement 


For 2003 and 2005, the parental involvement measures were calculated from the same five items 
(Exhibit 16). Teachers were asked to estimate the percentage of parents who participated in their 
schools through parent-teacher conferences (64 percent and 58 percent for 2003 and 2005, 
respectively), by demanding frequent reports (36 percent and 30 percent, respectively), by 
actively volunteering (16 percent and 12 percent, respectively), by observing classrooms (11 
percent and 10 percent, respectively), and by being active in PTA or PTO organizations (16 
percent and 14 percent, respectively). The 2007 parental involvement measure included four 
different items relating to parental involvement: school improvement plan involved parental 
involvement (79 percent), parents involved in selecting school reform (25 percent), school 
encourages parents to be involved in mission and goals (59 percent), and parents and community 
preferences were influential in classroom teaching (37 percent). In each year, nonmissing 
responses were averaged, and a school was credited as implementing the parental involvement 
measure if the average was equal to or greater than 0.4. 
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Exhibit 16 
Means and Standard Deviations of Survey Items Used to Construct the Parental 
Involvement Measure, by Year 


Standard 
Year Parental Involvement Items Deviation 


2003 | Parent-teacher conferences 0.26 
Demanding frequent reports 0.32 
Actively volunteering 0.17 
Observing classrooms 0.17 
Actively participate in PTO | 666 | 0.20 
Parent-teacher conferences 0.28 
Demanding frequent reports 0.23 
Actively volunteering 0.12 
Observing classrooms 0.12 


Actively participate in PTO 0.15 
2007 | School improvement plan covers parental involvement 0.41 
Parents involved in selecting school reform 0.43 


School encourages parents to be involved in mission 536 0.49 
and goals 


Parent and community preferences are influential in 536 F 0.48 
classroom teaching 


Exhibit highlights: Although survey items used to measure parental involvement were the same 
in 2003 and 2005, they changed in 2007. The reader may compute the means into a percentage 
by multiplying the mean value by 100. For example, the 2003 mean score of 0.64 indicates that 
respondents estimated that 64 percent of parents were active in their schools through parent- 
teacher conferences. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) Surveys. 


Case Study Findings 


Several types of parental involvement were evident from the case studies. Schools reported 
parent sign-off on homework as the most frequently used parent involvement strategy. They also 
reported use of parent resource centers and use of parent liaisons or parent coordinators. Several 
schools reported parent involvement in school advisory councils or committees. Instances of 
parents assisting in classes were reported only in two schools in the original group of 15 case 
study sites (encompassing 30 schools). 


A specific example of effective parent involvement involved a school in which parents helped 
with fund-raising events, ran an after-school program, and served as advocates at the district 
office for facilities and maintenance. Support was gained by use of a Communities in Schools 
program that linked families with services such as a migrant program, ESL classes, housing 
assistance, family violence prevention, and health services. In addition, the school sought 
bilingual staff members, parent representatives were elected annually to the school council, and 
parents had a voice in the school. Although a decline in parent attendance was noted during the 
second site visit, increasing participation was reported during the third visit. The increase in the 
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third year was reported to be the result of leadership exerted by a core group of parents. Parent 
volunteers were also reported to have provided adult ESL and computer training. 


All of the schools in the improving schools case study had some form of parent involvement 
initiatives. Seven had a parent resource center and a range of strategies for involving parents in 
their students’ learning, including homework sign-off, family nights, and family literacy. Two 
schools had full-time parent coordinators and an additional two schools had part-time 
coordinators. Finally, one school had the staff members visit all students’ homes. 


Component 8: External Assistance 


The external assistance measures constructed for 2003 and 2007 rely on different items than the 
2005 measure (Exhibit 17). Consequently, any year-to-year comparisons should be made with 
caution. In 2003 and 2007, the external assistance measure is constructed from six yes or no 
questions on the types of external assistance available. Nonmissing responses were averaged, and 
a school was credited as implementing the external assistance measure if the school reported that 
at least 80 percent of the nonmissing items were reported implemented. 


The 2005 survey includes four items that make up the external assistance measure. First, teachers 
were asked how many hours of training on reform they had received. The highest number 
reported was 959 hours. Some respondents may have interpreted this question as meaning over 
the life of the reform program, while others may have interpreted the question as the number of 
hours in the last year. Also, because there is often high teacher turnover in lower-performing 
schools, some respondents may have been in their school for only a portion of the CSR award 
implementation period. However, to scale this number to vary between 0 and 1, it was divided by 
1,000 and thus varies between 0 and 0.96. The next three questions (strategies include 
curriculum materials; all teachers received training on all strategies; and ongoing support is 
available) were asked up to four times, once for each reform identified in the school. The 
responses for each question were averaged to derive a school-level measure for each item that 
varied between 0 and 1. The external assistance measure was constructed by taking the average 
of the nonmissing four items, and schools with an average of at least 0.7 were counted as having 
implemented this item. 
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Exhibit 17 
Means and Standard Deviations of Survey Items Used to Construct the External 
Assistance Measure, by Year 


Standard 
Year External Assistance Items Deviation 


2003 


Hours (1,000) of training on reform strategy 
Strategies include curriculum materials 

All teachers received training on all strategies 
Ongoing support is available 


2007 | Onsite consulting 


Onsite consulting SSCS 
Professional development —=~=~“‘*‘*~*~dtCiSC*dYC=C‘ Ta 
Networking SSSCSC~C~*~“~*~*‘“~*~*~*~dtC GPSS 


Exhibit highlights: The external assistance measures constructed for 2003 and 2007 rely on 
different items than the 2005 measure. The reader may compute the means into a percentage by 
multiplying the mean value by 100. For example, the 2003 mean onsite consulting score of 0.70 
indicates that 70 percent of respondents indicated that onsite consulting from external entities 
was available. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) Surveys. 


Case Study Findings 


External consultants served as one source of assistance for schools. At one school, the reform 
model developer was reported not only to have provided classroom demonstration but also to 
have worked with another external facilitator to help teachers incorporate method strategies into 
action plans, assist with curriculum mapping, and help teachers recognize the link between the 
school improvement plan and classroom activities. In another example, a school received a 
National Science Foundation grant and used the money to fund one mathematics facilitator and 
one science facilitator to focus on closing the achievement gap for traditionally underachieving 
groups of students. These facilitators provided professional development throughout the course 
of the study. Other types of external support received by schools included funding (National 
Science Foundation and 21st Century Fund), assistance from regional labs, a business partner, 
outside experts, and support from universities. 
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Component 9: Evaluation 


The 2003 and 2005 measures for evaluation are nearly the same, but one item in the 2003 
measure (whether there is a formal written plan to evaluate progress) was not asked of 
respondents in 2005 (Exhibit 18). All of the questions asked in both years were yes or no 
questions. For each year, nonmissing responses were averaged, and a school was credited as 
implementing the evaluation component if the average score was 1.0 in 2003 and at least 0.75 in 
2005. This distinction was made because of the one item not present in the 2005 survey and the 
large drop in several of the items from 2003 to 2005. 


For 2007, the evaluation measure relies on only one item in which respondents were asked 
whether their school improvement plan included periodic evaluation goals. Eighty percent of the 
principals who responded indicated that their plan does have periodic evaluation goals. 


Case study results showed that no school reported that it evaluated whole-school reform. 
Therefore, no case study discussion is included for this component. 
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Exhibit 18 
Means and Standard Deviations of Survey Items Used to Construct the Evaluation 
Measures, by Year 


2007 


| 03 | 005 | OTT 
Standard Standard Standard 
Evaluation items __|_w_| Mean | Deviation | _N__| Mean | Deviation | _N_| Moan | Deviation | 
Does the school 
improvement plan have 
a mechanism for 


periodic evaluation 


[Evaluation Plan Topics [TT i 
FStudent performance | 711 | 0.96 | 0.20 | 678 | 1.00 007 | NA| NA | NA_| 


assistance 
Sources of financial aan 0.43 | 
support 


Is there a formal written 
plan to evaluate 712 | 0.85 NA NA NA NA 
progress 


Exhibit highlights: Although survey items used to construct the evaluation measure were largely 
the same in 2003 and 2005, they changed in 2007. The reader may compute the means into a 
percentage by multiplying the mean value by 100. For example, the 2003 mean score of 0.84 for 
whether the school improvement plan has a mechanism for periodic evaluation indicates that 84 
percent of respondents indicated that the plan did have such a mechanism. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) Surveys. 


Component 10: Coordination of Resources 


The 2003 and 2005 measures for the coordination of resources component are constructed from 
different measures (Exhibit 19), and any year-to-year comparisons should be made cautiously. 
All of the questions asked in both years were yes or no questions. For each year, nonmissing 
responses were averaged, and a school was credited as implementing the coordination of 
resources component if the average score was 1.0 in 2003 and at least 0.7 in 2005. This 
distinction was made because of the increase in the types of funds that were asked about in 2005. 
That is, some schools may not have received foundation grants or local donations and may have 
reported on their survey that they did not have control over these funds. 


In 2007, the coordination of resources measure relies on an item in which principals were asked 
whether their school had funding flexibility needed to implement the major elements of the 
reform at their school. Eighteen percent of principals responded that they definitely did have that 
flexibility. 
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Exhibit 19 
Means and Standard Deviations of Survey Items Used to Construct Coordination of 
Resources Measure, by Year 


Standard 

Year Deviation 
2003 | The school has control over budgetary issues 
The school has control over personnel decisions 


a 
coordinated 
[Align Tilelactviies —SSSC*dtSCTC«d?s=Ctw St 


The school has control over the following 
resources 


Exhibit highlights: The coordination of resources measure was constructed on different survey 
items each year. The reader may compute the means into a percentage by multiplying the mean 
value by 100. For example, the 2003 mean score of 0.77 for the school has control over 
budgetary issues indicates that 77 percent of respondents indicated that the school does have 
such control. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) Surveys. 


Case Study Findings 


There was evidence of district support and coordination in a majority of schools visited from the 
original 15 case study sites. In addition, a majority of schools qualified for support under Title I, 
although detail on how Title I services were coordinated with other aspects of the reform efforts 
was generally lacking. State support and coordination was less evident, however. 


There were a few examples where coordination across all levels was in place. In one instance, a 
school facilitator worked with the method developer to refine curriculum mapping while the state 
provided a review and approval of the reform model evaluation plan. District support and 
coordination was evident also during the second visit to this school. Finally, findings from the 
third site visit revealed that the school had used Title I funding to hire a consultant to work with 
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the school and coordinate with the district coaches to ensure that aspects of the reform model 
continued to be emphasized across the school. 


There were also examples of an absence of integration and coordination between the district and 
the state. In one case, the state was the driver not only of the CSR effort but also of reform 
efforts in non-CSR schools. The district, in contrast, used its own program improvement 
initiatives, and the CSR program was a low priority. Although there was no information as to 
how, or whether, this conflict was resolved, the ultimate reliance on state assessments and 
benchmarks for the accountability measure may have been the deciding factor. In another 
instance, the state tried to use the reform model as the umbrella for other reforms. In this district, 
the model did not match the district curricula. In the final year, the model was discontinued 
because the district reforms were considered robust. 


District decisions could also have a negative effect. In one instance, a district designated an 
improving school to serve as a “lighthouse” school that received students from other schools as 
part of the district’s choice plan. The district also added additional grades to the school, with 
little time for preparation, and the improving school subsequently lost its status because of low 
student performance. This school then became a sending school rather than a receiving school 
and may illustrate the fragile nature of some school improvement efforts. 


State frameworks in the form of school improvement plan expectations, grade-level content 
expectations, and core curriculum and professional development requirements supported reform 
and had an effect in at least two of the improving schools. In one instance, the state’s format for 
school improvement planning guided the school’s reform efforts for more than five years, with 
some adjustments. 


FINDINGS 


Were schools that received CSR awards more likely to implement the legislatively 
specified components of CSR than other schools? 


> No, consistent with earlier study findings, both CSR and non-CSR schools 
implemented a similar number of legislatively specified components in both 2003 
and 2007. The number of components rose slightly for CSR and non-CSR schools 
during this period at both the elementary and middle school levels. 


Both CSR and non-CSR schools implemented an average of four or fewer components in 2003 
and fewer than five in 2007 at both the elementary and middle school levels (Exhibit 20). Both 
CSR and non-CSR elementary schools reported making similar gains in implementing 
components in that the number of components implemented increased by 0.7 in CSR schools and 
0.8 in non-CSR schools. At the middle school level, CSR and non-CSR schools in both 2003 and 
2007 also reported similar increases in the number of components implemented. 
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Exhibit 20 
Average Number of CSR Components Implemented by CSR 
and Non-CSR Schools in 2003 and 2007 


Po Elementary Schools Middle Schools 


Change Change 
from 2003 from 2003 
School Type 2003 2007 to 2007 2003 2007 to 2007 
CSR Schools P39 [| 46 | om | 34 | 41 | os | 
Non-CSR Schools | 35 | 44 | oe | 32 | 42 | 10" | 


Difference Between CSR 


Exhibit highlights: In 2003, CSR elementary schools reported implementing a somewhat 
higher average number of components than non-CSR schools. No other significant differences 
between CSR and non-CSR schools were found. Both CSR and non-CSR schools reported 
increases between 2003 and 2007 in the number of components implemented. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and 
Outcomes (ECSRIO) surveys. 


Note: + p < .10; * p< .05; ** p< .01; N= 150 for CSR elementary schools; N = 124 for non- 
CSR elementary schools; N = 112 for CSR and non-CSR middle schools. Standard errors for 
elementary schools are included in Exhibit A.5; standard errors for middle schools are in 
Exhibit A.6. 


Although the average number of components reported implemented in CSR and non-CSR 
schools was similar, the distribution of components implemented indicates minor differences in 
comprehensiveness between CSR and non-CSR schools. Nearly similar percentages of CSR and 
non-CSR elementary schools implemented more than seven components in 2007 (Exhibit 21). At 
the middle school level, a higher percentage of non-CSR schools than CSR schools implemented 
between four and six components (Exhibit 22). 
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Exhibit 21 
Number of Components Implemented in CSR and Non-CSR 
Elementary Schools in 2007 


Number of 
Components 
Implemented 


ONon-CSR 
7 and Higher BCSR 


10% 20% 30% 40% 50% 


Percentage of Schools 


Exhibit highlights: A slightly higher percentage of Comprehensive School Reform (CSR) elementary 
schools reported implementing seven or more components in 2007 than did non-CSR schools. A 
slightly higher percentage of non-CSR schools than CSR schools implemented four to six 
components. The same percentage of CSR and non-CSR schools implemented zero to three 
components. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and Outcomes 
(ECSRIO) surveys. 
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Exhibit 22 
Number of Components Implemented in CSR and Non-CSR 
Middle Schools in 2007 


Number of 
Components 
Implemented 


ONon-CSR 
HCSR 


T T T T 


20% 30% 40% 50% 
Percentage of Schools 


Exhibit highlights: A slightly higher percentage of Comprehensive School Reform (CSR) middle 
schools implemented seven or more components in 2007 than did non-CSR schools. A slightly higher 
percentage of non-CSR middle schools reported implementing at least four components. The 
percentage of CSR schools implementing zero to three components was 4 percentage points higher 
than that of non-CSR middle schools. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and Outcomes 
(ECSRIO) surveys. 


Three reasons may explain the similar implementation gains in both CSR and non-CSR school 
reform models. First, as shown in three of the case study sites, a school’s receipt of a CSR award 
can allow its school district to shift resources away from that school and toward other non-CSR 
schools also in need of improvement. Second, NCLB was in its first full year in 2002-03; newly 
available assessment data may have prodded school districts to assist schools identified as low- 
performing. Third, as school districts gained experience with implementing the CSR program, 
they may have taken the lessons learned from those schools and applied them to non-CSR 
schools. This behavior was evident in two of the case study sites. 
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V. OVERALL STUDY CONCLUSIONS 


This study views the CSR program through numerous lenses, including achievement analyses, 
surveys of school reform implementation, and multiple case studies. This chapter summarizes the 
key findings deriving from this project and details what we see as the major implications for 
future federal education policy-making as well as for conducting high-quality research and 
evaluation studies in this area. 


SUMMARY OF FINDINGS 


In this summary of findings, we first lay out what we found about the implementation of the 
federal CSR program and related achievement outcomes. Specifically, we describe how states 
allocated their federal CSR program dollars and supported schools. We then detail the progress 
schools made in implementing their CSR awards by focusing on the implementation of the 
legislatively mandated components and the adoption of models with a scientific research basis. 
Finally, we describe achievement outcomes associated with the federal CSR program and its 
implementation. 


Our findings indicate that, overall, the federal CSR program did not lead to much comprehensive 
school reform or achievement gains. However, we also found that achievement gains were 
somewhat more likely in schools selecting models subsequently identified as having a scientific 
research base. The importance of adopting such models was generally more important in 
mathematics than in reading. 


Overall Results of the Federal Comprehensive School Reform Program 


The federal Comprehensive School Reform program did not yield comprehensively reformed 
schools nor was it associated with widespread achievement gains. There were some instances 


where we did observe achievement gains, however. These instances were somewhat more likely 
in mathematics in schools that selected models identified as having a scientific research base. 


In addition, through intensive case study investigations, we did find some instances where 
schools undertaking comprehensive school reform substantially improved their operations and 
subsequent achievement outcomes. After the discussion of the federal CSR program as a whole, 
we will turn to identifying the central conclusions from these case studies. We found that there 
was not a single route to improving low-performing schools; rather, success relied on the 
substantial overlap and interconnectedness among the many activities schools undertook as well 
as the alignment of those activities to specific contextual factors within the school. 


Implementation of the Federal Comprehensive School Reform Program 


CSR funds were strongly directed to high-poverty schools, i.e. those with poverty rates of at least 
75 percent, and those with high concentrations of minority students. Almost half (45 percent) of 
the CSR schools were high-poverty schools, nearly three times greater than the percentage of all 
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high-poverty schools (16 percent) and more than one and a half times the percentage of Title I 
schools (26 percent). 


Many states and districts gave priority for CSR funding to such schools.”4 One state, for 
example, listed all the schools identified in need of improvement and provided CSR funds to the 
most “in need,” working through the list until the funds ran out. Slightly more than half of the 
states awarded CSR funds based on a combination of merit and academic need. Significantly, 
when states evaluated CSR applications based solely on merit, fewer high-poverty schools 
received CSR awards. High-poverty schools may lack capacity to develop proposals, which was 
mitigated by including need as a criterion. 


Although states largely succeeded in providing CSR funds to those schools most in need, schools 
receiving CSR awards made little progress in implementing more than just a few of the 
legislatively mandated components and were largely indistinguishable from non-CSR schools 
that were similar in baseline achievement and demographics. By the third year of their grants, 
CSR schools reported implementing 4 of the 11 mandated components on average; two years 
later, they had not made much progress beyond that. These levels of implementation are quite 
similar to a group of comparison non-CSR schools that participated in this study, indicating low- 
performing, high-need schools (both CSR and non-CSR) were reforming at similar rates. 


Furthermore, only one-third of the schools receiving CSR awards selected reform models that 
would eventually be identified as having a scientific research basis. Schools primarily adopted 
externally developed models that lacked such a research base, or “home-grown,” locally 
developed models.?5 


This finding is similar to that of another major national study of comprehensive school reform, 
the National Longitudinal Examination of Comprehensive School Reform (NLECSR) (Aladjem 
et al., 2006), which studied implementation of seven of the most commonly adopted, externally 
developed CSR models. The NLECSR study also found that CSR schools had implemented the 
components of their CSR models at levels similar to non-CSR comparison schools and that both 
groups of schools had similar increases in fidelity of implementation over time. However, 
whereas the NLECSR found that implementation increased between the first and third years of 
model implementation, such increases were not evident in ECSRIO. The NLECSR also found 
that the degree of implementation was related to the model being implemented: those models that 
are considered highly prescriptive, for example, Success for All, tended to be implemented with 
greater fidelity than models that were not as prescriptive. ECSRIO did not focus on differences 


24. Findings on how states allocated their CSR funds to districts and schools derive from the Second-Year 
report, “Longitudinal Assessment of Comprehensive School Reform Program Implementation and Outcomes,” 
which is an internal document to the U.S. Department of Education, compiled by the Office of Planning, 
Evaluation, and Policy Development/ Policy and Program Studies Service, November 2007. 

25. Reports from the CSRQ Center, which formed the foundation for our measures of the research base of CSR 
models (see U.S. Department of Education, 2008, for a complete discussion of how research bases of CSR 
models were measured), were not available at the time that 2002 CSR awardees made their model selections; 
however, schools may have used data from two earlier reviews of CSR models: Herman (1999) and Borman, 
Hewes, Overman, & Brown (2002). The CSRQ reports drew heavily from the data used in previous reviews of 
CSR models. Of the 40 ratings conducted, 25 models were rated as having some evidence of effectiveness. The 
CSRQ Center relied on the number and quality of research studies of the models to develop its ratings. 
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between models and, as a result, may have missed this potentially important covariate of 
implementation. 


Three reasons may potentially explain the similar implementation gains in both CSR and non- 
CSR school reform models in our study. First, as shown in three of the case study sites, a 
school’s receipt of a CSR award can allow its school district to shift resources away from that 
school and toward other non-CSR schools also in need of improvement. Second, NCLB was in 
its first full year in 2002—03; newly available assessment data may have prodded school districts 
to assist all schools (CSR and non-CSR) identified as low-performing. Finally, as school districts 
gained experience with implementing CSR, they may have taken the lessons learned from those 
schools and applied them to non-CSR schools. This behavior was evident in two of the case 
study sites and was cited as a potential explanation in NLECSR. 


If these explanations are valid, then the federal CSR program may be responsible for more 
reform than is being given credit here. That is, it may have provided states and districts with the 
resources and information to assist a large number of schools identified as being in need of 
improvement, whether they were CSR schools or not. Conversely, other factors might explain 
the absence of difference in implementation of reforms between CSR and non-CSR schools in 
this study. The pressing need of assisting large numbers of schools may have prevented states 
and districts from supporting CSR schools that needed in-depth assistance. Further, with the 
accountability requirements in NCLB, many schools may have pursued strategies targeting 
assistance to those students nearest the state’s proficiency cut point as well as to specific 
population groups (such as English language learners). Such strategies might well have diverted 
attention from more comprehensive approaches to school reform. 


Achievement Outcomes Associated With the Federal Comprehensive School Reform 
Program 


The federal CSR program did not result in achievement gains. During the time that schools were 
implementing their awards, the 2002 cohort of CSR schools made small statistically significant 
gains in elementary mathematics and reading achievement. However, a group of comparison 
schools made similar gains, indicating that the CSR program was no better than comparable Title 
I schools at improving achievement after three years. Similarly, five years after initially 
receiving their CSR awards, schools receiving awards did not demonstrate larger achievement 
growth than matched comparison schools not receiving CSR grants. In fact, achievement gains 
were nonexistent in CSR elementary schools and lower than their comparison schools at the 
middle school level. 


When we examined the relationship between implementation and achievement, we found in the 
ECSRIO study that the number of components a school implemented after three years did not 
predict achievement gains. In contrast, the NLECSR study found a relationship between 
implementation and achievement. In that study, CSR schools that implemented their models with 
fidelity, especially between implementation years three and five, had higher achievement gains 
than other schools. The NLECSR also found model-specific achievement effects. Schools that 
adopted Success for All tended to implement it faithfully and thus realized achievement gains. 
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One potential explanation for the differences in study findings is the different foci of the two 
studies: specifically, NLECSR focused on fidelity to specific curriculum models while ECSRIO 
focused on fidelity to the principles in the comprehensive school reform law. The focus of 
ECSRIO on fidelity to the law, though appropriate for a program evaluation of a federal effort, 
may not capture those more nuanced aspects of implementation related to achievement.?° 


One instance where ECSRIO found some evidence of improved achievement is when CSR 
schools adopted models with a scientific research base. In general, the results demonstrated 
stronger relationships between the presence of models with scientific research bases and 
improved achievement in mathematics compared with those models and reading. Specifically, 
schools implementing reform models with some scientific research backing of effectiveness were 
more likely than others to experience achievement gains in middle school mathematics. In 
addition, low-performing schools that adopted CSR models with scientific evidence of 
effectiveness had a better chance of improving their elementary mathematics scores than schools 
adopting models without evidence of effectiveness. There was also some evidence, albeit 
weaker, that schools adopting models with some scientific evidence of effectiveness were more 
likely to experience achievement gains in middle school reading than schools adopting other 
models. 


Case Studies of School Improvement 


Despite the finding that the federal CSR program as a whole did not lead to improvements in 
achievement or school operations, we did find instances where targeted, sustained efforts 
resulted in achievement gains. Results reported in the Achieving Dramatic School Improvement: 
An Exploratory Study report (U.S. Department of Education, 2010) not only highlighted 
instances of achievement gains but also expanded the knowledge base about the challenges and 
intricacies of comprehensive school reform. That study of rapid improvement schools (i.e., those 
that made substantial achievement gains in a short time period) and of schools improving at a 
slower and steadier pace specifically identified leadership, school climate, instruction, and data 
use as important factors in the improvement process. 


School Leadership 


One of the most frequently cited practices for improving schools is changing the school leader. 
The arrival of a new principal sends a clear message that change is necessary. In line with the 
research literature, this study found that both the principal and distributed leadership were 
important in stimulating, implementing, and, in some cases, sustaining reforms to improve 
student achievement. That is, although we found instances where strong principals arrived in a 
low-performing school and almost single-handedly began the turnaround process, evidence 
emerged in many of our case studies that improving schools possess varying degrees of 
distributed leadership. This distributed leadership strategy appeared to be especially important in 
providing continuity when schools experienced turnover of a principal. 


26. The NLECSR study examined schools that were receiving and not receiving CSR funding. 
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Principals created conditions and structures to distribute leadership by mobilizing staff members 
and building a professional community—some with selected participants, others requiring 
universal participation. In all cases, principals tied those structures to school improvement 
efforts. For instance, several principals in the case study schools placed a major emphasis on 
enhancing the roles and responsibilities of school staff members for school improvement by 
recruiting more experienced staff members to participate in school decision making, opening up 
the decision-making process to the broader staff community (particularly when the previous 
involvement had been limited to a few members), and/or instituting committees or research 
teams that zeroed in on specific topics to support school improvement. 


Reported benefits of distributed leadership were interrelated and included shared responsibility, 
greater staff buy-in, more effective implementation of new practices, continuity of leadership, 
and enhanced collaboration. For example, by drawing on teachers from across grade levels, the 
organization of committees in one of the schools enhanced both horizontal (within grade) and 
vertical (across grades) collaboration. Teachers at another school believed that they were 
creating their own reforms, even though they were receiving considerable external help, because 
their own senior teachers were rolling out the strategies. 


Another benefit of distributed leadership is that it can be critical to maintaining continuity of 
leadership and thereby sustaining reforms despite leadership changes. At one school, for 
example, the staff members selected by the principal to serve on the management team continued 
to serve in pivotal leadership roles at the school five years later. When the principal left a school 
after the 2005-06 school year, its teacher leaders continued to lead and run the school with 
seemingly little guidance from the new principal. This case study information leads us to 
hypothesize that these schools’ continued achievement growth across two school leaders might 
be partially attributed to the continuity provided by a distributed leadership model.?’ 


School Climate 


The literature on school improvement and turnaround schools suggests that improving the 
climate in troubled schools can accompany—and facilitate—improved achievement (Herman et 
al., 2008). Changes to the climate that make the school safer and more orderly can serve a dual 
purpose: they demonstrate that it is possible to make quick and dramatic changes to the school, 
motivating staff members and students to support successful turnaround efforts (Herman et al., 
2008), and they eliminate a set of pressing, nonacademic needs to allow staff members to focus 
on academics (Kowal & Hassel, 2005). 


All of the schools visited in our study faced school climate challenges at the beginning of the 
improvement period, for example, a disruptive student learning environment or uninvolved 
parents, and each school adopted specific strategies intended to improve this condition. Nearly 
all of the schools considered changes to improve school climate as important, and in about half 
of the schools, new principals considered addressing school climate concerns as a necessary first 
step in laying the groundwork for instructional improvement efforts. Common approaches to 


27. For additional information, please see Chapter 3: Leadership from Achieving Dramatic School Improvement: An 
Exploratory Study (U.S. Department of Education, 2010). 
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improving school climate included enhancing behavior management efforts, involving parents 
meaningfully in school governance, and soliciting resources from the community.?° 


Instruction 


The literature on turning around low-performing schools suggests that a relentless focus on 
improving student achievement is a common success factor. All of the case study schools that 
made dramatic improvement reported making instructional improvements to raise student 
achievement. Common strategies to enhance instruction, in line with the research literature, 
included aligning the curriculum to district or state standards and assessments; adopting a new 
curriculum; and increasing learning time through an extended school year, after-school 
programs, or block scheduling within the school day. 


To support instructional improvement efforts, almost all schools reported sharing and 
systematically using data to guide instructional changes. One school, for example, prominently 
displayed data boards throughout the school and, instead of using them for punitive monitoring, 
established a shared responsibility for results, with one teacher noting “we all sink, or we all 
swim.” At another school, the notion of data display and transparency was extended to students, 
who regularly tracked their own progress against an “aim line” that served as a means of 
comparing their current achievement with established goals. 


Professional development from coaches and external consultants, often hired through district or 
state support, also appeared to be key in improving instruction. This strategy not only provided 
immediate instructional assistance but also laid the groundwork for schools becoming less reliant 
on funding and external assistance. Involvement in extensive professional development helped 
identify and develop teacher leaders who helped to sustain and institutionalize reforms.?? 


Federal and State Support 


In addition to actions at the school level, districts, states and the federal government can play an 
important role in stimulating both rapid improvement and more incremental achievement gains. 
We found that accountability requirements can push schools to change, and additional resources 
can support these efforts. Respondents in about half of the schools that were visited in the 
dramatic school improvement study cited district support, guidance and assistance as being 
instrumental to their success. Furthermore, four of these schools received substantial state 
support. While respondents were somewhat less likely to mention states and the federal 
government as specifically and directly affecting local reform, the overall accountability context 
appeared to have an important motivating effect at the local level. Respondents noted that the 
consequences for chronic low performance, as determined by federal and state accountability 
measures, had stimulated change. 


28. For additional information, please see Chapter 4: School Climate from Achieving Dramatic School 
Improvement: An Exploratory Study (U.S. Department of Education, 2010). 

29. For additional information, please see Chapter 5: Instructional Improvement Strategies from Achieving 
Dramatic School Improvement: An Exploratory Study (U.S. Department of Education, 2010). 
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Although the primary form of external support was financial, some respondents provided 
examples of in-kind assistance, predominantly from the districts. In-kind assistance was 
generally in the form of consultation and professional development for instructional coaches and 
teachers. In some cases, schools and districts used funding to purchase assistance directly from 
state and private agencies. Other forms of district support included assigning experienced 
principals with the explicit purpose of turning the school around or guiding reform efforts by 
establishing consistent expectations.?° 


Sustaining Improvement 


Each of the schools visited in the study faced challenges in sustaining their improvement 
strategies and achievement gains, including challenges caused by lost resources and staff 
turnover. Some respondents perceived their schools’ achievement gains as being related to the 
actions of, and resources provided by, districts, states, and, in some cases, the federal 
government. These respondents were concerned about the potential loss of external support. 
Some of these schools found ways to replace lost resources, and some have succeeded in 
institutionalizing improved instructional practices. 


Most schools developed strategies to alleviate resource fluctuations, for example, by seeking 
supplemental fiscal resources to replace lost funds and building expertise among staff members 
to reduce reliance on external support. Schools generally had an influx of resources over several 
years at a time. As grant funding streams ended, 8 of the 11 schools studied were able to rely on 
new funding streams. 


Several schools made themselves less reliant on funding and external assistance by providing 
extensive professional development for their teachers and by building teacher leaders through 
distributed leadership opportunities. Five schools used their funds and other resources to focus 
on enhancing the knowledge and skills of their staff members through professional development 
or to purchase or develop materials or instructional programs that would remain in place once 
funding ended. One school’s principal focused on securing grants that would build staff skills to 
outlive the life of the grant. Regional consultants at another school trained a selected group of 
teacher leaders on instructional practices and on training other teachers in those strategies. Near 
the end of our study, that school lost its school improvement specialist because of lack of 
funding. However, much of the implementation of the instructional practices had become 
institutionalized. 


Respondents also identified staff turnover, both voluntary and involuntary, as an impediment to 
sustaining academic achievement. One charter school consistently saw high staff turnover as 
teachers sought to move from that school to regular, higher paying, tenure-track positions in the 
local public school district. Another school reported nearly half of the teaching staff being new to 
the school in 2004—05, largely because of a new school opening up in the district. To combat the 
problems associated with staff turnover, a few schools strengthened the orientation of new staff 
members. At two schools, teachers participated in a new teacher academy. Yet another instituted 


30. For additional information, please see Chapter 6: External Support from Achieving Dramatic School 
Improvement: An Exploratory Study (U.S. Department of Education, 2010). 
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a mentor system that assigned new teachers to an experienced member of the staff to help 
integrate and acculturate the new teachers to the school. 


Closing Discussion: No One Way to Improve Low-Performing Schools 


One of the key findings from the Achieving Dramatic School Improvement: An Exploratory 
Study (U.S. Department of Education, 2010) report is that although there are several overarching 
categories of activity on which improving schools generally rely, the paths toward school 
improvement varied based on local needs and conditions. Some placed greater emphasis on one 
factor (e.g., distributed leadership) than another (e.g., transparent use of student-level data); 
others chose a different order of factors or they combined features of reform into unique ways to 
establish a coherent whole-school approach. Furthermore, reform strategies interacted in multiple 
ways, suggesting that the same reforms may be more or less successful depending on differences 
in leadership, staff capacity, community support and other factors. There is no “one way” to 
implement an effective school improvement program since different factors play out in different 
ways in different settings. 


This report also highlighted the multiple ways by which schools can improve quickly and 
dramatically. The NLECSR findings similarly illustrated the interdependence of CSR model 
components. Neither this study nor NLECSR identified a single path to improvement or a unique 
bundle of components or activities associated with school improvement. On the contrary, both 
studies demonstrated that sustained improvement implies substantial overlap and 
interconnectedness among the many activities that schools undertake. These suggested multiple 
connections lead directly to the question of how the federal government can foster scalable or 
systemwide reform that resemble the successes observed in our case study sites and in NLECSR. 


We would posit that systemwide reform or scaled-up improvement of this type requires highly 
contextualized and resource-intensive support. The U.S. Department of Education would need to 
begin to provide substantial amounts of direct and individually tailored technical assistance to 
state departments of education and possibly also to local school districts. The assistance the 
Department would provide to states would mirror the situationally dependent assistance some 
states in the case studies provided districts and schools. 


The Department could expand the scope of work of its network of regional Comprehensive 
Centers to provide this kind of support to states and districts attempting to turn around low- 
performing schools. Currently, the Comprehensive Centers are tasked with providing assistance 
primarily to state education agencies to implement NCLB; to use scientifically valid teaching 
methods and assessment tools; and to facilitate communication between education experts, 
school officials and parents. A new expanded mission for these Centers could take the form of 
developing and providing training to assist states, districts and schools in identifying the 
organizational approach or approaches (e.g., leadership changes, accountability systems, and/or 
the implementation of research-based school reform models with fidelity) needed to improve 
quality instruction in schools. 


Chapter V 56 


FUTURE RESEARCH NEEDED FOR IMPROVING CHRONICALLY LOW-PERFORMING 
SCHOOLS 


Future research for improving chronically low-performing schools could play a significant role 
in shaping the nature of subsequent federal efforts and of other service providers so they are 
targeted on interventions most likely to result in positive effects. To do so, we recommend that 
such an agenda address the following critical needs. 


1. The need for a more robust national database. In our national study of the federal CSR 
program, CSR award databases provided limited information about the CSR model chosen 
beyond the initial year of funding. These databases also lacked information about the total 
amount of funding provided beyond the first year of the award. 


Similarly, one challenge in selecting sites for the case studies of dramatic school improvement 
was the absence of up-to-date national data on student achievement. Reliable estimates of 
school-level performance should ideally be derived from longitudinal, student-level data to 
ameliorate problems inherent in “percent proficient” school-level calculations through the use of 
robust student growth curve models. This issue is of particular importance in schools that 
experience high levels of student mobility, which is the case in many low-performing schools. 


We were also constrained in our work by having a data set that not only was out of date but also 
provided achievement information only at the school level. Because the school-level data 
available from the NLSLSASD were not updated, we needed to gather data from individual 
states that varied in their completeness and in how recent they were. 


Although one might reasonably expect future updates to national school-level data sets such as 
the NLSLSASD, it seems unlikely that a comparable national database of student-level data will 
be compiled in the near future. Nevertheless, the U.S. Department of Education could work with 
the states to enhance the EdFacts database by including relevant activities within the schools. 
Critical data elements should address principal and faculty stability, measures of school climate, 
reform models) in use, award amounts from federal funds and other major sources, use of 
extended hours for instruction, use of data, and instructional support such as teacher mentors and 
internal and/or external coaches. Furthermore, the Department should make these data available 
to the research community in real time so that timely, relevant analyses can be conducted. 


2. The need for real-time investigations and the avoidance of retrospective analyses. 
Researchers, policy-makers, and practitioners in recent years have paid much attention to the 
scientific search for “what works” to improve schools and turn them around. Our study’s 
findings underscore the challenges inherent in such efforts by drawing attention to the fact that 
comprehensive school reform is not just about adopting a set of effective or promising practices. 
It is also about navigating and adapting to a constantly changing landscape that needs to be 
monitored on an ongoing basis. Real-time feedback and analysis is needed to accomplish this 
type of reform. 


With a more robust database as described above, researchers could identify potential low- 
performing schools that turned around as soon as dramatic improvements in achievement were 
observed. As schools are identified as potential case study sites, researchers could identify 
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matched comparison schools. Next, with potential treatment and comparisons identified, 
researchers could quickly begin more in-depth, qualitative work to examine and track practices 
close in time to each school’s achievement improvement. Such an approach not only would 
minimize the error induced by retrospective data collection but also would provide critically 
important counterfactual data from the identified comparison sites. 


Finally, after tracking these matched pairs of successful turnaround and comparison schools for 
several years, enough should have been learned about the process of dramatic school 
improvement to design one or several interventions that would be the basis for experimental or 
quasi-experimental studies of particular attributes of successful school turnaround identified in 
the first two stages of this study. 


This ability to design interventions could also suggest an alternative, expanded role for the 
Regional Educational Laboratories, whose goal is to provide “access to high-quality, 
scientifically valid education research through applied research and development projects, 
studies, and other related technical assistance activities.”3! The Regional Educational 
Laboratories could be charged with identifying and conducting real-time investigations of low- 
performing schools that dramatically improved in their regions that could be used to develop 
interventions with descriptions of the conditions under which those interventions seem to work 
best. 


3. The need for well-developed implementation measures. One of the key differences between 
the NLECSR and ECSRIO is in the definition of implementation. By focusing on fidelity to the 
law, ECSRIO did not capture differences in models and essentially treated all models as having 
the same components. However, the findings from the dramatically improving schools case 
studies and from the NLECSR highlight the fact that school reform is context-specific. That is, 
while certain overarching themes should be addressed (e.g., leadership, school climate and 
instruction), the specifics of addressing these critical areas rely on the particular circumstances of 
the school. 


Thus, what is needed are well-developed school reform implementation measures serving two 
purposes that are at risk of potentially conflicting with each other. First, implementation 
measures need to be specific enough to effectively characterize the school reform effort. 
Effective leadership occurs not just through the presence of an effective leader but, rather, 
evolves through the correct style and distribution of leadership in a school to match that school’s 
context. Second, implementation measures must be broad enough to allow for comparison across 
school contexts so that research can effectively identify the set of practices that a school can 
adopt in a particular set of circumstances and feel reasonably confident that those practices have 
been shown in previous research to yield positive reform. 


Once these measures are developed, it would not be unreasonable to expect states, districts and 
schools receiving funding for school reform to provide periodic robust information about 
implementation. The systematic collection and analysis of such data would be a major step 


31. See the welcome statement of the U.S. Department of Education’s Institute of Education Sciences, 
Regional Education Laboratory Program, which is found on the program’s home page at 
http://ies.ed.gov/ncee/edlabs/. 
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toward a continuously more refined and comprehensive understanding of the implementation 
phenomena and how they relate to improved teaching and learning in classrooms. This in-depth 
understanding would in turn help policy-makers design better interventions over time, leading to 
more effective reforms for low-achieving schools and heightened student achievement. 
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APPENDIX A. 
STANDARD ERROR TABLES FOR ANALYSES 


This appendix includes the standard error tables for the analyses presented throughout the report. 


STANDARD ERRORS FOR REPORT EXHIBITS 


Exhibit A.1 
Standard Errors for Differences Between the 2002 Cohort of Title | CSR Schools and 
Title | Non-CSR Schools 


Non-CSR CSR 
Standard Standard 
Error Error 


Standardized Assessment Scores 


Elementary Mathematics 
Elementary Reading 
Middle School Mathematics 
Middle School Reading 
Membership 
Percentage Minority 
Percentage FRL 48,872 0.122 1.096 


Exhibit highlights: Standard errors for estimates presented in Exhibit 5 in the main body of the 
report. 


Source: Common Core of Data (CCD). 


Note: “FRL” refers to students eligible for free or reduced-price lunches. 
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Exhibit A.2 
Standard Errors for Differences Between the 2002 Cohort of Title | CSR Schools and 
Non-CSR Title | Schools Chosen by Propensity Scoring: Schools With Elementary 
Mathematics and Reading Achievement 


Non-CSR CSR 
Standard Standard 
Error Error 


Standardized Assessment Scores 


Elementary Mathematics 0.042 0.063 


Elementary Reading 0.041 0.059 
Membership 11.266 17.670 


Percentage Minority 1.599 2.324 


Percentage FRL 1.002 1.338 


Exhibit highlights: Standard errors for estimates presented in Exhibit 6 in the main body of the 
report. 


Source: Comprehensive School Reform (CSR) Awards Database; Common Core of Data 
(CCD); National Longitudinal School-Level State Assessment Score Database (NLSLSASD). 


Note: “FRL” refers to students eligible for free or reduced-price lunches. 


Exhibit A.3 
Standard Errors for Differences Between the 2002 Cohort of Title | CSR Schools and Non- 
CSR Title | Schools Chosen by Propensity Scoring: Schools With Middle School 
Mathematics and Reading Achievement 


Non-CSR CSR 


Standard Standard 
N 
Error Error 


Standardized Assessment Scores 


Middle School Mathematics 0.053 95 0.082 


Middle School Reading 0.048 95 0.093 


Membership 26.716 109 43.026 


Percentage Minority 2.283 109 3.516 


Percentage FRL 1.442 103 2.287 


Exhibit highlights: Standard errors for estimates presented in Exhibit 7 in the main body of the 
report. 


Source: Comprehensive School Reform (CSR) Awards Database; Common Core of Data (CCD); 
National Longitudinal School-Level State Assessment Score Database (NLSLSASD). 


Note: “FRL” refers to students eligible for free or reduced-price lunches. 
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Exhibit A.4 
Standard Errors for Changes in Standardized Assessment Scores in CSR and Non-CSR 
Schools From 2002-03 to 2006—07 


CSR Non-CSR 


x Schools Schools 


Elementary Mathematics 560 0.043 0.041 
Elementary Reading 563 0.038 0.040 
Middle School Mathematics 291 0.040 0.041 
Middle School Reading 294 0.045 0.045 


Exhibit highlights: Standard errors for estimates presented in Exhibit 8 in the main body of the 
report. 


Source: Comprehensive School Reform (CSR) Awards Database; Common Core of Data (CCD); 
National Longitudinal School-Level State Assessment Score Database (NLSLSASD). 


Appendix A 67 


Exhibit A.5 
Standard Errors for Average Number of CSR Components Implemented by 
CSR and Non-CSR Elementary Schools in 2003 and 2007 


CSR Schools (N = 153) 0.136 0.156 
Non-CSR Schools (N = 128) 0.143 0.162 


Exhibit highlights: Standard errors for estimates presented in Exhibit 20 in the main body of the 
report. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and Outcomes 
(ECSRIO) surveys. 


Exhibit A.6 
Standard Errors for Average Number of CSR Components Implemented by CSR and 
Non-CSR Middle Schools in 2003 and 2007 


Non-CSR Schools (N = 55) 0.262 0.259 


Exhibit highlights: Standard errors for estimates presented in Exhibit 20 in the main body of the 
report. 


Source: Evaluation of the Comprehensive School Reform Program Implementation and Outcomes 
(ECSRIO) surveys. 
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APPENDIX B. 
DATA COLLECTION INSTRUMENTS 
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Principal Inventory Year 1 


Longitudinal Assessment of 
Comprehensive School Reform 


PRINCIPAL INVENTORY 


Paperwork Burden Statement 

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 
information unless such collection displays a valid OMB control number. The valid OMB control number for this 
information collection is 1875-0222. The time required to complete this information collection is estimated to 
average 20 minutes per response, including the time to review instructions, search existing data resources, gather the 
data needed, and complete and review the information collection. If you have any comments concerning the 
accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of 
Education, Washington, D.C. 20202-4651. If you have comments or concerns about the contents of this 
questionnaire, write directly to: WestEd. 
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INVENTORY OF SCHOOL REFORM EFFORTS 
Dear Principal, 


WHO IS CONDUCTING THIS INVENTORY? 

WestEd and COSMOS Corporation, educational research organizations under contract to the 
U.S. Department of Education, request your participation in this inventory. WestEd and the 
Department of Education are conducting this inventory by the authority of Section 1607 of the 
No Child Left Behind Act of 2001 (P.L. 107-110). 


WHAT IS THE PURPOSE OF THIS INVENTORY? 

This inventory 1s designed to understand how reform at your school changes the ways that you 
and your staff approach teaching and learning. This inventory focuses on changes that can affect 
every facet of school operations including classrooms, professional development, schoolwide 
operations (e.g. scheduling) and management, and administration of the school building. 


HOW WILL THE RESULTS BE USED? 

The data from this inventory will be used by the U.S. Department of Education and Congress to 
evaluate the implementation and effectiveness of federal school reform legislation. Data will be 
reported only in statistical summaries; your individual responses will be kept confidential. 


WHY SHOULD YOU PARTICIPATE IN THIS INVENTORY? 

We are conducting this inventory with only a sample of schools. Therefore, the value of your 
individual contribution is greatly increased because it represents many other schools. We 
encourage you to participate in this voluntary inventory. 


WHERE SHOULD YOU MAIL YOUR COMPLETED QUESTIONNAIRE? 

Please return your completed questionnaire in the enclosed envelope. If you do 

not have the return envelope, or have other inquiries, call toll free 1-866-880-2773 or email 
csr(@duerrevaluation.com. You may also mail your questionnaire to: 


WestEd 

Attn: John Flaherty 

4665 Lampson Avenue 
Los Alamitos, CA 90720 


THANK YOU FOR YOUR COOPERATION IN THIS IMPORTANT EFFORT! As soon as we 
receive your questionnaire, remuneration for you or your school will be mailed immediately. 


OMB No. 1875-0222 Approval Expires 01/31/2006 
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Instructions: 


The questions on this inventory apply to reform being implemented at your school. All questions refer to 


reform occurring at your school during the last school year (2002-2003). 


Please refer to the state of reform at your school at the end of last year when answering each question. 
Some questions ask that you mark the one best response, while others ask you to mark all that apply. 


I. School Planning 

1. How formal is the school improvement plan at your school? Last year (2002-03) 
(mark one response) 

a. Comprehensive written plan o 

b. Outline ofa plan 0 

c. Written mission statement only jo 

d. No formal plan but teachers generally share the same ideas o 

e. Multiple plans o 

f. Noplan (Skip to question 4) 0 
2. What aspects of reform are covered by the school improvement plan? Last year (2002-03) 


(mark all that apply) 
a. Measurable goals or objectives 
b. Mechanism for periodic evaluation of goals 
c. Curriculum and instruction content 
d. Student assessment rubrics 
e. Classroom management guidelines 
f. Professional development activities 
g. Parental involvement plan 
h. Framework for participation in school management 
i. Integration of new technology 
j. Other (specify) 


Qogoaaoaoaagaaoaaudd 


OMB No. 1875-0222 Approval Expires 01/31/2006 
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3. What factors influenced the content of your formal school improvement 
plan? 


State or district content standards 


os 


State or district performance standards 
Needs identified through a school needs assessment 


c 
d. School performance standards 


@ 


Specifications of adopted/adapted reform design 


rh 


Assigned by district/state 
g. Other (specify) 


4. Has your school been identified as a low performing school? 


a. Yes 
b. No 


5. Has your school been sanctioned because of low performance? 


a. Yes 
b. No 


II. Reform Characteristics 


Last year (2002-03) 


(mark all that apply) 
o 


guagaaadg 


Last year (2002-03) 
(mark one response) 
0 
o 


Last year (2002-03) 
(mark one response) 


Oo 
o 


6. How is reform at your school designed? 


a. Totally designed at this school 

b. Adapted with modifications from external source 

c. Adapted selected parts from multiple external sources 
d. Adopted unmodified from external source 


7. Indicate the primary designer for the reform efforts at your school: 


a. Locally developed 

b. School district 

c. University 

d. Private developer or publisher 


e. Other (specify) 


Last year (2002-03) 
(mark one response) 


o 


Qo Oo 


Last year (2002-03) 
(mark one response) 


o 


QgQagaqda 
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8. Does the primary reform effort at your school have a name? 


a. Yes 
No 


c. Don’t know 


9. Ifso, write the name here: 


10. Is your school currently implementing more than one reform? 


a. Yes 
No 


c. Don’t know 


11. If so, list the names here: 


12. At which grade levels in your school are reforms mainly focused? 


Last year (2002-03) 
(mark one response) 
0 
5 


o 


Last year (2002-03) 
(mark one response) 
0 
0 


0 


Kk} ODO @ @ @ & &@ & B&B 2) Gd 2 Gmarkall that apply) 


13. Which categories best describe the focus of reform efforts at your school? 


Whole-school 


Reading/language arts 


sp 


Mathematics 
Science 

Social studies 
Arts 

Other (specify) 


mo 2 6 


ga 


14. Who is primarily responsible for selecting the school reform in which your 
school participates? 


School board 

District central office 
School administrators 
School improvement team 


Teachers 


ho nego Bp 


Parents 
Other (specify) 


ga 


Last year (2002-03) 
(mark all that apply) 
o 


QgQagaaqgaaoqgd 


Last year (2002-03) 


(mark all that apply) 
o 


ogoaaqad0audnodu 
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15. Which of the following factors contribute to the selection of reform at your 


school? 
a. Cost of reform 
b. Theoretical or research foundation 
c. Successful implementation at other schools 
d. “Fit” with school needs 
e. State or district mandate 
Compatibility with assessment tools 
g. Best published results 
h. Don’t know 


—e 


16. Does your school have control over budgetary issues at the school site? 


a. Yes 
b. No 


17. Does your school have control over personnel decisions at the school site? 


a. Yes 
b. No 


III. Faculty Role in Reform 


Other (specify) ee, 


Last year (2002-03) 


(mark all that apply) 


o 


gQ0oagaaaodad da 


Last year (2002-03) 
(mark one response) 


o 
o 


Last year (2002-03) 
(mark one response) 


o 
qo 


18. Did the faculty formally vote to adopt the current reform in your school? 


a. Yes 
No 


c. N/A No adopted reform 


19. Estimate the percentage of teachers who participate in reform at your school 


(write percentage in space at right): 


20. What factors determine the extent of teacher participation in reform at your 


school? 


All teachers participate at this school 


os 


Our reform is subject-specific 


c. Our funding limits the number of teachers who can participate 


A. 


Our school reform is being phased in over time 


e. Other reason (specify) 


Last year (2002-03) 
(mark one response) 


0 
0 


0 


Last year (2002-03) 


(mark all that apply) 
og 


gogogddo 
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21. What evidence shows that the reform chosen at your school improves student 


achievement? 
a. Independent research 
b. Research conducted by reform designer 


Research that included comparison or control groups 
Results from students at this school 
Other (specify) 


Don’t know 


mo 2a 6 


22. Does your school have performance goals for students at the end of each 
school year? 


a. Yes, by grade level 
b. Yes, by content area 
c. Other (specify) 

d. No 


23. Does your school have intermediate student performance goals within each 
school year to gauge student progress? 


a. Yes, by grade level 
b. Yes, by content area 
c. Other (specify) 
d. No 


24. What factors influenced the creation of performance goals at your school? 


a. State content standards 
b. State testing requirements 
c. Local standards or testing requirements 
d. Postsecondary eligibility requirements 

e. Reform efforts 

f. School priorities 
g. Parent concerns 
h. Other (specify) 


i. None, our school does not have performance goals 


Last year (2002-03) 


(mark all that apply) 
Oo 


Qoagaqaun 


Last year (2002-03) 


(mark one response) 


o 


Qgagda 


Last vear (2002-03 


(mark one response) 


0 


QgQagu 


Last year (2002-03) 
(mark all that apply) 
o 


QOoaaoanouvdaogduduv.dvng 
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25. Which academic subjects are covered by goals or benchmarks at your school 


; Last year (2002-03 
for student achievement? 


(mark all that apply) 


a. Reading/language arts o 
b. Mathematics go 
c. Science o 
d. Social Studies 0 
e. Arts 0 
f. Other (specify) o 


26. Does your school have a formal test preparation program to prepare 


Last year (2002-03) 
students for mandated state tests? 


(mark one response) 


a. Yes o 
b. No (Skip to question 29) Oo 


27. Ifso, which grade levels participate in the test preparation program? (mark all that apply) 


Kk} DO @ 8 @ & &@ & B&B @) Go Gs (markall that apply) 


28. Ifso, how much time do students spend in the test preparation program Last vear (2002-03 


each year? 
(mark one response) 
a. 2 days or less o 
b. Between 2 and 5 days o 
c. Between 5 and 10 days o 
d. More than 10 days o 


V. Evaluation 


29. Does your school have a formal written plan to evaluate progress toward 


2 Last year (2002-03) 
improvement goals? 


(mark one response) 


a. Yes o 
b. No o 
30. Which of the following factors influence the direction of evaluation? Last year (2002-03) 
(mark all that apply) 
a. School-developed requirements o 
b. Requirements of reform o 
c. District requirements og 
d. State requirements qo 
e. Federal Title I requirements o 
f. Other (specify) qo 
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31. What topics are addressed in the formal evaluation plan? Last year (2002-03 
(mark all that apply) 
Student performance o 


oS 


Program implementation 
Parental participation 
Staff development 


Utility of external assistance 


mh ® 2 © 


Sources of financial support 
Other (specify) 


OQOoagaaqgag 


ge 


VI. Support for School Reform 


32. Does the state reward districts or schools for meeting student achievement 
Last year (2002-03) 


goals? 
(mark one response) 
a. Yes o 
b. No o 


33. Does the state sanction districts or schools for failing to meet student 


, . Last year (2002-03) 
achievement goals? 


(mark one response) 
a. Yes o 
b. No o 


34. Which sources of funding contribute to implementation and operation of 


ss a . Last year (2002-03) 
reform efforts at your school? 


(mark all that apply) 


a. Federal CSR funds o 
b. Title I o 
c. Other federal funds 0 
d. Special state grants o 
e. Discretionary district funds o 
f. Foundation grants o 
g. Local community or business donations Oo 
h. Other (specify) o 
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35. How have existing resources been coordinated to support reform efforts at 


Last year (2002-03) 
your school? 


(mark all that apply) 


Alignment of district professional development 0 
b. Alignment of Title I activities 0 
c. Alignment of other special funds (e.g. bilingual education, or magnet o 
schools) 
d. Reallocation of staff positions o 
e. Other (specify) o 


36. Which one of these external entities is primarily responsible for supporting 
Last year (2002-03) 
reform efforts at your school? (choose only one) 


(mark one response) 


University o 


Ss 


Regional Education Laboratory 
State agency 
School district 


Reform program developer 


rm ® 210 


Teachers from other schools 
Non-affiliated consultants 
Other (specify) 


None, our school does not use external support 


= a 


gooaogo0nuduauddcadnonad 


es 


37. What kind of assistance does this primary external entity provide to 


Last year (2002-03) 
support reform efforts at your school? 


(mark all that apply) 


a. Onsite consulting o 
b. Professional development 0 
c. Networking opportunities o 
d. Written materials for students 0 
e. Written materials for teachers 0 
f. Software or technological assistance 0 
g. Other (specify) o 
h. None, our school does not use external support 0 
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38. What type of additional support has been available through your district 
for school reform efforts? 


ao 5 B 


Pr @ 


= 09 


Administering a needs assessment 

Providing additional school staff to support school reform 
Selecting a school reform model 

Writing grants to support school reform 

Providing professional development for school reform 
Release time for teachers 

Other (specify) 

None, the district does not supply additional support 


39. What type of additional support has been available through your state for 
school reform efforts? 


sf 


rm 


Administering a needs assessment 

Selecting a school reform model 

Writing grants to support school reform 

Providing professional development for school reform 
Release time for teachers 

Other (specify) 


None, the state does not supply additional support 


VIL. Instructional Practice and Professional Development 


40. What teacher enhancement opportunities are available to teachers in your 
school? 


r>oaas p 


Mentoring new teachers 

Coaching other teachers 

Making management decisions 

Making decisions related to curriculum development 
Participating on grade-level teams 

Participating on content area teams across grades 
Other (specify) 


Last year (2002-03) 


(mark all that apply) 
o 


QgQoagoagaouaed 


Last year (2002-03) 


(mark all that apply) 
o 


gooagaagda 


Last year (2002-03) 


(mark all that apply) 
o 


gogageaqgda 
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41. Are teachers scheduled to have a common planning period during the day? Last year (2002-03) 
(mark all that apply) 


a. Yes, by grade level oO 
b. Yes, by content area a) 
c. Other (specify) o 
d. No, no common planning time is scheduled qo 
42. Who organizes the scope and sequence of the curriculum at your school? Last year (2002-03) 
(mark all that apply) 
a. State o 
b. District 0 
c. School o 
d. Individual teachers 0 
43. How is professional development organized at your school? Last year (2002-03) 
(mark all that apply) 
Teachers select activities sponsored by the district or other organizations o 
b. The district identifies common professional development themes for all o 
teachers 
c. The school reform plan includes professional development activities for o 
all teachers 
d. All teachers participate in the same professional development activities q 
e. Each teacher has an individual professional development plan o 


44. Which kinds of professional development opportunities are available to 


Last vear (2002-03 
teachers at your school? 


(mark all that apply) 


Reading/language arts instruction o 
Mathematics instruction o 
c. Instructional strategies for low-achieving, limited-English-proficient, im) 


special education, and/or migrant students 


d. Ensuring that curriculum and instruction are consistent with state and/or o 
district content standards 


e. Ensuring that curriculum and instruction are consistent with state and/or o 
district assessments 


f. Implementation of a school reform model 


Monitoring individual students’ progress toward learning goals 


pm a 


Interpreting reports of student achievement data 
Other (specify) 


QgoQoagdg 


_ 
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45. In what format or setting is professional development offered to teachers at 


Last year (2002-03 
your school? 


(mark all that apply) 


a. Seminars ms) 
b. Workshops o 
c. Study groups o 
d. Make and take o 
e. Regional and national conferences o 
f. Courses for college credit o 
g. Other (specify) o 
h. None, our school does not provide formal professional development o 


46. How is time allocated to enable teachers to participate in formal 


P oge Last year (2002-03) 
professional development opportunities? 


(mark all that apply) 
Pupil-free days og 
Release time 


Faculty meeting time 


aes f 


Personal time (evenings and weekends) 


e. Holidays (including summer vacation) 


gQoagdada 


f. Our school does not provide formal professional development 


47. Does your school provide the equivalent of over 10 days of professional 


Last year (2002-03) 
development per year? 


(mark one response) 
a. Yes o 
b. No o 


VIII. Role of Parents 


48. By what means does your school formally communicate with parents? Last year (2002-03) 
(mark all that apply) 

a. Designated parent coordinator o 
b. Regular newsletter 

c. Telephone calls 

d. Email 

e. Website 

f. In the language spoken at home (other than English) 

g. Other (specify) 


h. No formal communication with parents 


QgQoOauaaead 
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49. In what ways does your school encourage parents to be involved in 
governance? 


ao B 


rm @ 


> a 


Defining school mission and goals 
Choosing instructional materials 
Hiring teachers and staff 
Scheduling and school calendar 
Evaluating school improvement 
Fundraising 
Other (specify) 


School governance does not involve parents 


50. Estimate the percentage of parents who are engaged with the school in the 
following ways (write percentage in space at right): 


Ss 


mh @ 2 04 


Attend parent-teacher conferences 

Demand frequent reports on their children’s progress 

Actively volunteer 

Observe classroom activities 

Actively participate in a formal parent organization (PTA/PTO) 
Other (specify) 


IX. Principal Characteristics 


Last year (2002-03) 


(mark all that apply) 
o 


gQoaoaoaad 


Last year (2002-03) 


51. How many years have you been a principal? 


52. How many years have you been a principal at this school? 
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Are any of the following design(s) currently in use at your school? (mark all that apply) 


Entire-School 
O) Accelerated Schools 
© America’s Choice 
O ATLAS Communities 
O Audrey Cohen College: Purpose-Centered Education 
O Center for Effective Schools 
Child Development Project 
©) Coalition of Essential Schools 
O Community for Learning 
O Co-nect 
Core Knowledge 
Different Ways of Knowing 
Direct Instruction 
Edison Schools 


Expeditionary Learning Outward Bound 


High Schools That Work 

High/Scope Primary Grades Approach to Education 
Integrated Thematic Instruction 

MicroSociety® 

Modern Red Schoolhouse 

Montessori 

Onward to Excellence 

Paideia 

QuEST 

Roots & Wings 


School Development Program (Comer) 

Success for All 

Talent Development HS with Career Academies 
Talent Development Middle School 

The Learning Network 

Turning Points 

Urban Learning Centers 

Ventures Initiative and Focus® System 

Other whole-school reform 


QOO0O0O000) ONONONOROROR0O0OO0O0 00000 


QOO0O000noonoo0 


QQO0O00o00 


QoOOQ QO 


QOO0O00no0no0oo0u02) 


Reading/Language Arts 
Breakthrough to Literacy 

Carbo Reading Styles Program 
CELL/ExLL 

CORE 

Early Intervention in Reading 
Exemplary Center for Reading Instruction 
First Steps 

Junior Great Books 

Literacy Collaborative 

National Writing Project 

Reading Recovery 

Other reading/language arts design 


Mathematics 

Connected Mathematics Project 

Core Plus Mathematics Project 

Growing with Mathematics 

Interactive Mathematics Program 

MATH Connections® 

U. of Chicago School Mathematics Project 
Other mathematics design 


Science 

Developmental Approaches to Science, Health and 
Technology 

Foundational Approaches in Science Teaching 
GALAXY Classroom Science 

Iowa Chautauqua Program 

Other science design 


Other 

ACCESS 

Basic Skill Builders 

COMP: Creating Conditions for Learning 
Feuerstein’s Instrumental Enrichment 
HOSTS 

HOTS 

Lightspan Achieve Now 

Positive Action 

Responsive Classroom 
Success-in-the-Making 

Other - not listed on this page 


Thanks for your participation! As soon as we receive your questionnaire, remuneration for you or 


your school will be mailed immediately. 
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Principal Survey Year 2 


Longitudinal Assessment of 


Comprehensive School Reform 


PRINCIPAL SURVEY 
2004-2005 School Year 


Hae 
Gs a 
ERE 

es Soh ee 


According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information 
unless such collection displays a valid OMB control number. The valid OMB control number for this information collection 
is 1875-0222. The time required to complete this information collection is estimated to average 20 minutes per 
response, including the time to review instructions, search existing data resources, gather the data needed, and 
complete and review the information collection. If you have any comments concerning the accuracy of the time 
estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 
20202-4651. If you have comments or concerns about the contents of this questionnaire, write directly to: 


WestEd, attn: John Flaherty, Project Coordinator, 4665 Lampson Avenue, Los Alamitos, CA 90720 
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Dear Principal, 


WHO IS CONDUCTING THIS SURVEY? 

WestEd and Cosmos Corporation, educational research organizations under contract to the U.S. 
Department of Education, request your participation in this survey. WestEd and the Department 
of Education are conducting this survey by the authority of Section 1607 of the No Child Left 
Behind Act of 2001 (P.L. 107-110). 


WHAT IS THE PURPOSE OF THIS SURVEY? 

This survey is designed to understand how comprehensive reform at your school changes the 
ways that you and your staff approach teaching and learning. This survey focuses on changes 
that can affect every facet of school operations including classrooms, professional development, 
school-wide operations (e.g. scheduling) and management, and administration of the school 
building. 


HOW WILL THE RESULTS BE USED? 

The data from this survey will be used by the U.S. Department of Education and Congress to 
evaluate the implementation and effectiveness of federal school reform legislation. Data will be 
reported only in statistical summaries; your individual responses will be kept confidential. 


WHY SHOULD YOU PARTICIPATE IN THIS SURVEY? 

We are conducting this survey with only a sample of schools. Therefore, the value of your 
individual contribution is greatly increased because it represents many other schools. We 
encourage you to participate in this voluntary survey. 


WHERE SHOULD YOU MAIL YOUR COMPLETED QUESTIONNAIRE? 

Please return your completed questionnaire in the enclosed envelope. If you do 

not have the return envelope, you can mail the survey to the address below, or contact the study 
team via email or using our toll-free number provided below. 


Longitudinal Assessment of Comprehensive School Reform 
Attn: Susan Cragle 

55 Hanover Lane 

Chico, CA 95973 

1-866-880-2773 

csr@duerrevaluation.com 


THANK YOU FOR YOUR COOPERATION IN THIS IMPORTANT EFFORT! 
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Adequate Yearly Progress 


1. Has your school been identified as having failed to make adequate 


yearly progress (AYP) as defined by the accountability provisions of (mark one response) 
NCLB during the 2003-04 school year? 
a. Yes Oo 
b. No (skip to question 3) | 
2. How many consecutive years has your school failed to meet AYP? years 


School Improvement Plan 


3. Does your school improvement plan cover the following? (mark all that apply) 
a. Measurable goals or objectives O 
b. Mechanism for periodic evaluation of goals O 
c. Curriculum and instruction content O 
d. Student assessment rubrics i 
e. Classroom management guidelines O 
f. Professional development activities O 
g. Parental involvement plan |a 
h. Framework for participation in school management O 
i. Integration of new technology O 
j. Other (specify) LC 
k. No school improvement plan O 

4. What factors influenced the content of your formal school (mark all that apply) 

improvement plan? 
a. State or district content standards O 
b. State or district performance standards | 
c. Needs identified through a school needs assessment 4 
d. School performance standards 0 
e. Specifications of adopted/adapted reform design O 
f. Assigned by district/state 0 
g. Other (specify) O 
h. No school improvement plan C 
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School Reform 


The following questions refer to the most important reform strategy or strategies in your school. Such 
reform strategies are those intended to improve school operations and student outcomes. 


5. Please name the most important reform strategy or strategies in use at your school. (The table 
at the end of the survey will ask for more descriptive information about each strategy.) 


6. At which grade levels in your school is reform mainly focused? 


kK) OW 4) Bl td (ul) (mark all that apply) 


7. In what ways does your school evaluate progress toward school 
ae . (mark all that apply) 
reform goals? 
a. Student performance 
b. Program implementation 
Parental participation 
Staff development 


Utility of external assistance 


me 209 


Sources of financial support 
Other (specify) 


No evaluation plan at our school 


Oi CO ee Ce Ce 


> oe 
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Teacher Participation in Reform 


8. To what extent do you believe reform at your school: (Circle one number in each row.) 


Toa 


bles To some fairl ts 
Not at all small . y great 
extent large 
extent extent 
extent 
a. helps improve student learning 1 2 3 4 5 
b. helps teachers to teach more effectively I 2 3 4 5 
c. detracts from more important efforts 1 2 3 4 5 
d. improves communication among teachers l 2 3 4 5 
e. helps to improve student behavior 1 2 3 4 5 
9. What factors limit teacher participation in reform at your school? (mark all that apply) 
a. No limitations - all teachers participate at this school O 
b. Our reform is subject-specific — only teachers in a specific subject participate i 
c. Our funding limits the number of teachers who can participate | 
d. Our school reform is being phased in over time CI 
e. Other reason (specify) O 
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Professional Development 


This question refers to all activities intended to help teachers develop and improve their content 
knowledge and classroom instruction. Examples of professional development activities include mentoring 


programs and coaching in addition to more traditional activities such as internships, workshops, 
conferences, institutes, and college courses. These may be conducted within the school or outside the 
school setting. 


10. How is professional development organized at your school? Include 
only workshops, coursework, and conferences sponsored by your 
school, district, or state during the 2003-04 school year (including 
summer 2004). 


(mark all that apply) 


a. Teachers select activities sponsored by the district or other organizations 
b. The district identifies common professional development themes for all teachers 


c. The school reform plan includes professional development activities for all 
teachers 


d. All teachers participate in the same professional development activities 


oC) eee C) 


e. Each teacher has an individual professional development plan 


Parent Involvement 


11. In what ways does your school encourage parents to be involved in 


Parr ieee (mark all that apply) 
governance-related activities? 


Defining school mission and goals 


Ss 


Choosing instructional materials 


Hiring teachers and staff 


a 


Scheduling and school calendar 


& 


Evaluating school improvement 


rh 


Fundraising 
Other (specify) 


School governance does not involve parents 


OBOBOROs 


> 9p 
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Support for Reform 


12. Over which of the following resources do you have control at the 
school site for implementation and operation of reform efforts at (mark all that apply) 
your school? 


Federal Comprehensive School Reform funds 
Title I 
Other federal funds 


Special state grants 


Ss 


o Be 


i 


Discretionary district funds 


rh 


Foundation grants 
Local community or business donations 
Other (specify) 


OOOO00d0 


e 


13. What type of support has been available through your district for 


(mark all that apply) 
school reform efforts? 


Administering a needs assessment 


Ss 


Providing additional school staff to support school reform 


Selecting a school reform model 


o 


Writing grants to support school reform 


i 


Providing professional development for school reform 


Lar) 


Release time for teachers 
Other (specify) 
None, the district does not supply support 


OBORORO es 


pe a9 


14. Which of the following external entities supports reform efforts at 


(mark all that apply) 
your school? 


University 


S 


Regional Education Laboratory 
State agency 


o 


Reform program developer 


Teachers from other schools 


th Oo 


Non-affiliated consultants 
Other (specify) 


None, our school does not use external support 


Oe CO i OC i Ce 


p 99 
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Reform Strategies 


Schools often use single or multiple reform strategies designed locally or by external program developers. Some 
strategies are externally designed, for example Success for All or Co-nect. Other strategies are local and may not have a 
formal name. On this page we are collecting specific information about each strategy in use at your school. Begin by 


writing the name of the most important reform strategy at the top of the first column and then answer each question 
considering that strategy only. Then, using the next column, repeat the process again for each additional strategy at 
your school. If your school is using a single strategy, complete the first column only. If your school has no reform 
strategy, write “none” in the first column. 


Reform Strategy Name 9 FIFI IIDD> 


Description of Reform Strategy 


Does the strategy target curricular areas for 
improvement? 
If yes, indicate curricular area(s) targeted: 
(Circle all that apply.) 


Does the strategy target school organization and 
management for improvement? 

Does the strategy target community and parent 
involvement? 

Do you have evidence that this strategy improves 
student achievement? 

Does the evidence include comparison groups? 

Has the strategy been demonstrated to improve student 
achievement at other schools? 


Status of Reform Strategy 


Are all grades included in the strategy? 

Is this strategy being phased-in? 

Is the strategy accompanied by benchmarks for 
implementation? 

Did your district mandate this strategy? 

Will sanctions result from failure to implement this 
strategy? 


Support Provided through Reform Strategy 


Was this strategy developed externally? 

Does the strategy include specific curricular materials? 
Do guidelines for classroom practice accompany the 
strategy? 

How many hours of training on the strategy did 
teachers receive from the program developer? 

How many hours of training on the strategy did 
teachers receive from the district? 

How many hours of training on the strategy did 
teachers receive from the state? 

How many of these total training hours (from the 
developer, district and state) occurred at your school? 
Did all teachers at your school receive training to use 
the strategy? 

Is ongoing support for the strategy available through an 
onsite facilitator? 


(circle one) 


Yes No 


Language Arts 
Mathematics 


Social Studies 
Science 
Arts 
Other 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 

hours 
hours 
hours 
total 
hrs. 
Yes No 
Yes No 


(circle one) 


Yes No 


Language Arts 
Mathematics 


Social Studies 
Science 
Arts 
Other 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 

hours 
hours 
hours 
total 

hrs. 
Yes No 
Yes No 


(circle one) 


Yes No 
Language Arts 
Mathematics 
Social Studies 


Science 
Arts 
Other 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 

hours 
hours 
hours 
total 
hrs. 
Yes No 
Yes No 


(circle one) 


Yes No 


Language Arts 
Mathematics 


Social Studies 
Science 
Arts 
Other 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 
Yes No 

hours 
hours 
hours 
total 
hrs. 
Yes No 
Yes No 


OMB No. 1875-0222 


Approval Expires 01/31/2006 
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Principal Survey Year 2 


Background 


15. How many years have you been a principal? years 


16. How many years have you been a principal at this school? years 


THANK YOU FOR PARTICIPATING! 


Please return your completed questionnaire in the enclosed envelope. If you do 
not have the return envelope, you can mail the survey to the address below, or contact the study 
team via email or using our toll-free number provided below. 


Longitudinal Assessment of Comprehensive School Reform 
Attn: Susan Cragle 

55 Hanover Lane 

Chico, CA 95973 

1-866-880-2773 

csr@duerrevaluation.com 


OMB No. 1875-0222 Approval Expires 01/31/2006 
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Principal Survey Year 5 


Longitudinal Assessment of 


Comprehensive School 
Reform 


Principal Survey 


2006-07 School Year 


According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless 
such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1875- 
0222. The time required to complete this information collection is estimated to average 40 minutes per response, including the 
time to review instructions, search existing data resources, gather the data needed, and complete and review the information 
collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, 
please write to: U.S. Department of Education, Washington, D.C. 20202-4651. If you have comments or concerns about the 
contents of this questionnaire, write directly to: 


WestEd, attn: John Flaherty, Project Coordinator, 4665 Lampson Avenue, Los Alamitos, CA 90720 
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Dear Principal, 


WHO IS CONDUCTING THIS SURVEY? 

WestEd and Cosmos Corporation, educational research organizations under contract to the U.S. 
Department of Education, request your participation in this survey. WestEd and the Department 
of Education are conducting this survey by the authority of Section 1607 of the No Child Left 
Behind Act of 2001 (P.L. 107-110). 


WHAT IS THE PURPOSE OF THIS SURVEY? 

This survey is designed to understand how comprehensive reform at your school changes the 
ways that you and your staff approach teaching and learning. This survey focuses on changes 
that can affect every facet of school operations including grade-level planning, curriculum and 
instruction, data driven decision making, professional development and alignment with state and 
district reform efforts and priorities. 


HOW WILL THE RESULTS BE USED? 

The data from this survey will be used by the U.S. Department of Education and Congress to 
evaluate the implementation and effectiveness of federal school reform legislation. Responses to 
this data collection will be used only for statistical purposes. The reports prepared for this study 
will summarize findings across the sample and will not associate responses with a specific 
district or individual. We will not provide information that identifies you or your district to 
anyone outside the study team, except as may be required by law. 


WHY SHOULD YOU PARTICIPATE IN THIS SURVEY? 

We are conducting this survey with only a sample of schools. Therefore, the value of your 
individual contribution is greatly increased because it represents many other schools. We 
encourage you to participate in this voluntary survey. 


WHERE SHOULD YOU MAIL YOUR COMPLETED QUESTIONNAIRE? 

Please return your completed questionnaire in the enclosed envelope. If you do not have the 
return envelope, you can mail the survey to the address below, or contact the study team via 
email or using our toll-free number provided below. 


Longitudinal Assessment of Comprehensive School Reform 
Attn: Susan Cragle 

55 Hanover Lane 

Chico, CA 95973 

1-866-880-2773 


scragle@duerrevaluation.com 


THANK YOU FOR YOUR COOPERATION IN THIS IMPORTANT EFFORT! 
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Grade-Level and Content Planning 


a 


On average during the last 12 months, how many hours per week 
have you spent with others in common planning time? (Write in the 


number of hours) hours per week 


2. Who participates in these planning activities? (Circle one number in each row) 
Once ora Once or Once or Daily or 


fewtimes _twicea twice a almost 
Never a year month week daily 
a. Classroom teachers 1 2 3 4 5 
b. Teachers on special assignment 1 2 3 4 5 
c. Curriculum coaches 1 2 3 4 5 
d. School administrators 1 2 3 4 5 
e. Teachers from other schools 1 2 3 4 5 
f. District officials 1 2 3 4 5 
g. State officials 1 2 3 4 5) 
h. Reform program developers 1 2 3 4 5 
i. Other (specify) 1 2 3 4 5) 


Coordinated Curriculum and Instruction 


3. How much does each of the following influence what is taught in the classroom? (Circle one 
number in each row) 


Strong Somewhat Little or Somewhat Strong 
negative negative no positive positive 
influence influence influence influence influence 

a. State curriculum frameworks or content 
1 2 3 4 5 
standards 
b. District curriculum frameworks or content 
1 2 3 4 5 
standards 
c. Textbook/instructional materials 1 2; 3 4 > 
d. State testing requirements 1 2 3 4 5 
e. District testing requirements 1 2 3 4 3) 
f. Students’ special needs or individual 
: : 1 2 3 4 5 
instructional plans 
g. Parent or community preferences 1 2: 3 4 5) 
h. Professional development experiences 1 Z 3 4 5 
i. Diagnostic or classroom assessment results 1 2; 3 4 5 
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4. Which entities — state, district or school officials — are 
involved in decision-making for each of the following? = (mark ail that apply) 


State District School 
a. Selecting textbooks and other instructional materials [el ol L 
b. Selecting content, topics, skills to be taught [i] L] Fl 
c. Selecting teaching techniques ina Iai oO 
d. Creating student ability groups for instruction i] (3) le 
e. Allocating instructional time for each academic subject LJ L] L 
f. Determining the content of in-service professional development gO gO Og 
in which teachers participate 
g. Hiring new full-time teachers Ol oO L 
h. Planning school budgets L] L] L 
i. Determining professional and teaching assignments [el El L 
j. Establishing the school curriculum CJ CL] L] 
k. Other (specify) LJ L] i 


5. Has your school received any of the following designed to align 


curriculum and instruction with state and/or district content 
standards? (mark all that apply) 


Detailed standards-based curriculum guides 
Detailed pacing schedules 


c. Classroom-embedded assessments with a standard scoring rubric to monitor 
progress toward district or state standards 


d. Curriculum map aligning the required curricula and instructional programs to 
standards 


e. Curriculum map aligning the required curricula and instructional programs to 
assessments 


f. Model lesson plans based on the standards 


g. Analytic reports summarizing student achievement data to identify specific 
strengths and weaknesses related to the attainment of standards 


h. Information on listservs related to standards 


i. Professional development designed to ensure that curriculum and instruction are 
consistent with state and/or district content standards 


j. Professional development designed to ensure that curriculum and instruction are 
consistent with state and/or district assessments 


k. Other (specify) 


lt fF fim t- 
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Data Driven Decision Making 


6. Does your school receive reports or a summary of assessment results 


from state or district tests? (mark one response) 
Yes O 
No O 


7. Does your school receive any of the following to support the use of 
assessment results? aticalt Mate! 


a. Test results disaggregated by classroom [na 
b. Test results disaggregated for special populations of students LO 
c. Test results disaggregated by grade | 
d. Test results disaggregated by sub-test LJ 
e. Item-by-item review of test results im 
f. School reports showing trends over three years or more L] 
g. Training and consultation for school staff to allow school-based analysis of O 

student achievement data 

8. _ Did you receive state or district assessment results summarized by 

any of the following categories of students? hiarkalt thataoph 
a. Race or ethnicity LJ 
b. Gender L] 
c. Title I participation oO 
d. Migrant status LI 
e. Poverty status L] 
f. Limited English Proficiency (LEP) status LC] 
g. Students Individualized Education Programs (IEPs) L 


9. Who at your school receives such reports or summaries of 
assessment results? hieeeall shaban 


a. Principals/administrators (i 
b. Teachers | 
c. Parents oO 
d. Students im 
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10. Have you received any assistance or training on how to interpret or 
analyze state or district assessment results? ui ail tide asaiyh 


Yes, from the state 
Yes, from the district 


Yes, from another external entity 


a9 Fp 
Oo 


No assistance or training on interpreting or analyzing assessment results 


11. Which data sources are available to you and your school for student 


assessment or planning purposes? (mark all that apph 
a. “Raw” student performance data on state or district tests 


Reports summarizing student performance on state or district tests 


© 


Reports that analyze or interpret student performance on state or district tests 
based upon demographic characteristics (such as race or gender) or educational 
program (such as results for students with special needs) 


Reviews or evaluations of program implementation by external entity 
Diagnostic tests to measure student readiness 
Classroom observations by peers or administrators 


Reviews of student projects or portfolios 


= 0 -m © 


Surveys of students, parents, teachers or staff regarding the school climate and 
satisfaction 


fy 6 hue 


i. Research reports from external resources on the efficacy of improvement 
strategies 


j. Research conducted at the school (e.g., teacher or administrator action research 
projects; staff analysis of school data) on the efficacy of current improvement 
strategies 


k. Tests of student knowledge or skills at the end of instructional units 


1. Tests of student knowledge and skills at regularly scheduled times (e.g., 
quarterly or every six weeks) 
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12. To what extent have state or district assessment results 


been used for the following purposes? (stele Gre Habe) 


Little or no Moderate 


Great use 
use use — 
a. Measuring student progress toward meeting state or district 
1 2; 3 
standards or benchmarks 
b. Identifying students for intervention or support services 1 2 3 
c. Placement of students into remedial or advanced courses 1 D} 3 
d. Identifying areas where more professional development is 1 5 3 
needed 
e. Assessing teacher performance 1 2 3 
f. Helping teachers align their instruction to the school curriculum 1 2 3 
g. Establishing priorities for future instruction (e.g., more 1 > 3 
emphasis on subjects or topics where students scored low) 
h. Revising the school improvement plan (SIP) 1 2 3 
i. Evaluating school initiatives/programs 1 2) 3 


School Improvement Plans 


13. How formal is the school improvement plan at your school? 


(mark one response) 
Comprehensive written plan 


Outline of a plan 
Written mission statement only 
No formal plan but teachers generally share the same ideas 


Multiple plans 


me aos ps 
OO ROe 


No plan (Skip to question 18) 


14. Does your school improvement plan cover the following? 


(mark all that apply) 
Measurable goals or objectives 


a 
b. Mechanism for periodic evaluation of goals 

c. Curriculum and instruction content 

d. Student assessment rubrics 

e. Classroom management guidelines 

f. Professional development activities 

g. Parental involvement plan 

h. Framework for participation in school management 


i. Integration of new technology 


OBO OB OBO 


j. Other (specify) 
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15. What factors influenced the content of your formal school 
improvement plan? (mark all that apply) 


a. State or district content standards L 

b. State or district performance standards L 

c. Needs identified through a school needs assessment | 

d. School performance standards LJ 

e. Specifications of adopted/adapted reform design O 

f. Assigned by district/state L] 

g. Other (specify) zl 

16. Which entities — state, district or school officials — are 
involved in decision-making for each of the following? (nark ail that apph 
State District School 

Developing the school improvement plan oO O O 

b. Reexamining the school improvement plan LC LF LF 

c. Providing data to review for the school improvement plan (e.g., a a a 
standardized test data) 

d. Provide training on developing and/or reexamining the school gO gO gO 
improvement plan 

e. Establishing goals and benchmarks for the school O O O 

17. How often does your school review or monitor progress toward the 

school improvement plan’s goals? idvicoherEsponie 

a. Never (| 

b. Once every two years im 

c. Annually oO 

d. Two or three times a year im 

e. Quarterly O 

f. Monthly LJ 

g. Other (specify) LJ 
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Professional Development 


The question in this section refers to all activities intended to help teachers develop and improve their 
content knowledge and classroom instruction. Examples of professional development activities include 
mentoring programs and coaching in addition to more traditional activities such as internships, 
workshops, conferences, institutes, and college courses. These may be conducted within the school or 
outside the school setting. 


18. Howis professional development organized at your school? Include 
only workshop, coursework, and conferences sponsored by your 
ties : 6 pply) 
school, district, or state during the last 12 months. mark all that appl 


Teachers select activities sponsored by the district or other organizations 
b. The district identifies common professional development themes for all teachers 


c. The school reform plan includes professional development activities for all 
teachers 


d. All teachers participate in the same professional development activities 


of ff 


e. Each teacher has an individual professional development plan 


Adoption of Scientifically Based Methods 


19. Is your school using a school reform strategy or any combination of 


school reform strategies? (mark one response) 

Yes L 

No L 
c. Not sure LO 


20. 
Please name the most important reform strategy or strategies in use at your school. 


21. In what year did your school begin implementing the most 
important reform model or strategy? year 
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22. 


At which grade levels in your school is reform mainly focused? 
oa Bg we fF & @ & @ LL Mtark alt that appiy) 


23. 
How is reform at your school designed? (mark one response) 
a. Totally designed at this school CL] 
b. Adapted with modifications from external source L] 
c. Adapted selected parts from multiple external sources L 
d. Adopted unmodified from external source LJ 
24. 
Indicate the primary designer for the reform efforts at your school: — (mark one response) 
a. Locally developed L] 
b. School district oO 
c. University L] 
d. Private developer or publisher L] 
e. Other (specify) LJ 
25. 
Which categories best describe the focus of reform efforts at your (mark all that apply) 
school? 
a. Whole-school L 
b. Reading/language arts im 
c. Mathematics (ea 
d. Science L 
e. Social studies O 
f. Arts CL 
g. Other (specify) LJ 
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26. 


Who is responsible for selecting the school reform in which your (mark all that apply) 


school participates? 
School board 


District central office 


Se 


School administrators 
School improvement team 
Teachers 
Parents 

Other (specify) 


fo mo 2/19 
=) ee Le Le 


27. 


Which of the following factors contribute to the selection of (neh UL EERE CED) 
reform at your school? 


Cost of reform 
Theoretical or research foundation 


Successful implementation at other schools 


2 eS 


“Fit” with school needs 

State or district mandate 
Compatibility with assessment tools 
Best published results 

Don’t know 
Other (specify) 


= 9m - Oo 
ie Le eC 


rm 
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Capacity and Support for Reform 


The following questions refer to the most important reform strategy or strategies in your school. Such 
reform strategies are those intended to improve school operations and student outcomes. 


28. 
Which of these external entities is primarily responsible for (taricone response) 
supporting reform efforts at your school? 
a. University L] 
b. Regional Education Laboratory L] 
c. State agency im 
d. School district CL] 
e. Reform program developer | 
f. Teachers from other schools L 
g. Non-affiliated consultants oO 
h. State-designated assistance providers CL] 
i. Other (specify) LI 
j. None, our school does not use external support LO 


29. 


What kind of assistance does this primary external entity provide 
to support reform efforts at your school? 


Onsite consulting 


(mark all that apply) 


Ss 2 


Professional development 


Networking opportunities 


a | 


Written materials for students 
Written materials for teachers 
Software or technological assistance 
Other (specify) 


None, our school does not use external support 


OO Oe oe 


= 0 sm oO 


30. 


Was there a vote at your school to adopt or implement the current 
reform at your school? If so, who participated in the voting? 


Administrators 


Teachers 


(mark all that apply) 


Se 


c. Students 


Q 


Parents 


“ff 


e. None, there was no vote on the reform 
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31. 


Do you willingly participate in reform-related activities at your (ar One reponse) 
school? 

Yes CL] 

No L 


32. To what extent do you believe reform at your school: (Circle one number in each row) 


Toa 
foe To some fairl Fos 
Not at all small heed great 
extent large 
extent extent 
extent 
a. Helps improve student learning 1 2 3 4 5) 
b. Helps teachers to teach more effectively 1 2 3 4 5 
c. Detracts from more important efforts 1 2 3 4 2) 
d. Improves communication among teachers 1 2 3 4 5 
e. Helps to improve student behavior 1 2 3 4 5 
33. 
What factors limit teacher participation in reform at your school? —_ (mark all shat apply) 
a. No limitation — all teachers participate at this school LI 
b. Our reform is subject-specific — only teachers in a specific subject participate CL] 
c. Our funding limits the number of teachers who can participate O 
d. Our school reform is being phased in over time LF 
e. Other reason (specify) (ea 
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34. To what extent do you disagree or agree with the following statements about the academic 
program at your school? (Circle one number in each row) 
Strongly To some Strongly 


. Disagree Agree 
disagree 8 extent g agree 


a. At this school, we have a common 
understanding of the objectives we’re trying to 1 2 3 4 5 
achieve with students 


b. Goals and priorities for the school are clear 1 2 3 4 5 


c. Most teachers at this school have values and 


philosophies of education similar to my own ] 2 : i : 
d. Most of my colleagues share my beliefs and 
values about what the central mission of the 1 2 3 4 5 
school should be 
e. Most staff in this school share a focused 1 > 3 4 5 


vision for student learning 


f. Teachers are committed to developing 
partnerships with parent(s)/guardian(s) for 1 2 3 4 5 
student learning 

g. The school encourages parents to be involved 
in defining the school mission and goals 


35. To what extent does your school have the resources 
needed to implement the major elements of the reform 


at your school? (circle one number per row) 


Not Somewhat _ Definitely 
available available available 
Design team materials to support instruction 1 2 3 
Availability of design team materials to further implement the 1 > 3 
design in your school 
c. Professional development for teachers 1 2) 3 
d. Time for planning, collaboration, and development 1 2 3 
Staff or consultants to mentor, advise, and provide ongoing 1 ) 3 
support 
f. Technology and connectivity 1 2 3 
g. Funds 1 2 3 
h. Funding flexibility 1 2 3 
i. Support from staff 1 2 3 
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Background 


36. How many years have you been a principal? years 
37. How many years have you been a principal at this school? years 


THANK YOU FOR PARTICIPATING! 


Please return your completed questionnaire in the enclosed envelope. If you do 
not have the return envelope, you can mail the survey to the address below, or contact the study 
team via email or using our toll-free number provided below. 


Longitudinal Assessment of Comprehensive School Reform 
Attn: Susan Cragle 

55 Hanover Lane 

Chico, CA 95973 

1-866-880-2773 

csr@duerrevaluation.com 
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Teacher Inventory Year 1 


Longitudinal Assessment of 
Comprehensive School Reform 


TEACHER INVENTORY 


Paperwork Burden Statement 

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 
information unless such collection displays a valid OMB control number. The valid OMB control number for this 
information collection is 1875-0222. The time required to complete this information collection is estimated to 
average 20 minutes per response, including the time to review instructions, search existing data resources, gather the 
data needed, and complete and review the information collection. If you have any comments concerning the 
accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of 
Education, Washington, D.C. 20202-4651. If you have comments or concerns about the contents of this 
questionnaire, write directly to: WestEd. 
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INVENTORY OF SCHOOL REFORM EFFORTS 
Dear Teacher, 


WHO IS CONDUCTING THIS INVENTORY? 

WestEd and COSMOS Corporation, educational research organizations under contract to the 
U.S. Department of Education, request your participation in this inventory. WestEd and the 
Department of Education are conducting this inventory by the authority of Section 1607 of the 
No Child Left Behind Act of 2001 (P.L. 107-110). 


WHAT IS THE PURPOSE OF THIS INVENTORY? 

This inventory is designed to understand how reform at your school changes the ways that you 
and your staff approach teaching and learning. This inventory focuses on changes that can affect 
every facet of school operations including classrooms, professional development, schoolwide 
operations (e.g. scheduling) and management, and administration of the school building. 


HOW WILL THE RESULTS BE USED? 

The data from this inventory will be used by the U.S. Department of Education and Congress to 
evaluate the implementation and effectiveness of federal school reform legislation. Data will be 
reported only in statistical summaries; your individual responses will be kept confidential. 


WHY SHOULD YOU PARTICIPATE IN THIS INVENTORY? 

We are conducting this inventory with only a sample of schools. Therefore, the value of your 
individual contribution is greatly increased because it represents many other schools. We 
encourage you to participate in this voluntary inventory. 


WHERE SHOULD YOU MAIL YOUR COMPLETED QUESTIONNAIRE? 

Please return your completed questionnaire in the enclosed envelope. If you do 

not have the return envelope, or have other inquiries, call toll free 1-866-880-2773 or email 
csr(@duerrevaluation.com. You may also mail your questionnaire to: 


WestEd 

Attn: John Flaherty 
4665 Lampson Avenue 
Los Alamitos, CA 90720 


THANK YOU FOR YOUR COOPERATION IN THIS IMPORTANT EFFORT! As soon as we 
receive your questionnaire, remuneration for you or your school will be mailed immediately. 


OMB No. 1875-0222 Approval Expires 01/31/2006 
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Instructions: 


The questions on this inventory apply to reform being implemented at your school. All questions refer to 
reform occurring at your school during the last school year (2002-2003). 


Please refer to the state of reform at your school at the end of last year when answering each question. 
Some questions ask that you mark the one best response, while others ask you to mark all that apply. 


I. Teacher Characteristics 


1. How many years have you been a teacher? 


2. How many years have you taught at this school? 


3. How many years have you taught the same grade/subject area? 


4. What type of credential(s) do you hold? Last year (2002-03) 
(mark all that apply) 
a. Professional o 
b. Preliminary o 
c. Emergency o 
d. Substitute o 
e. Other o 
II. School Planning 
5. How formal is the school improvement plan at your school? Last year (2002-03) 
(mark one response) 
a. Comprehensive written plan 0 
b. Outline of a plan o 
c. Written mission statement only 0 
d. No formal plan but teachers generally share the same ideas 0 
e. Multiple plans ) 
f. Noplan (Skip to question 8) o 


OMB No. 1875-0222 Approval Expires 01/31/2006 


Appendix B 116 Teacher Inventory Year | 


6. What aspects of reform are covered by the school improvement plan? Last year (2002-03) 
(mark all that apply) 


a. Measurable goals or objectives ) 
b. Mechanism for periodic evaluation of goals 

c. Curriculum and instruction content 

d. Student assessment rubrics 

e. Classroom management guidelines 

f. Professional development activities 

g. Parental involvement plan 

h. Framework for participation in school management 
i. Integration of new technology 

j. Other (specify) 


QgQoagaaaaudd 


7. What factors influenced the content of your formal school improvement Last vear (2002-03 


plan? 
(mark all that apply) 
a. State or district content standards o 
b. State or district performance standards 0 
c. Needs identified through a school needs assessment o 
d. School performance standards 0 
e. Specifications of adopted/adapted reform design ao 
f. Assigned by district/state 0 
g. Other (specify) o 
8. Has your school been identified as a low performing school? Last year (2002-03) 
(mark one response) 
a. Yes o 
b. No qo 
9. Has your school been sanctioned because of low performance? Last year (2002-03) 
(mark one response) 
a. Yes 0 
b. No i) 


OMB No. 1875-0222 Approval Expires 01/31/2006 
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III. Reform Characteristics 


10. Does the primary reform effort at your school have a name? Last year (2002-03) 
(mark one response) 

a. Yes im) 

No o 

c. Don’t know o 


11. If so, write the name here: 


12. Is your school currently implementing more than one reform? Last year (2002-03) 
(mark one response) 

ay GR qo 

b. No o 

c. Don’t know 0 


13. If so, list the names here: 


14. At which grade levels in your school are reforms mainly focused? 


Hoe Be YY @ @ BB @ A GD ©  Cmark alt that appiy) 


15. Which categories best describe the focus of reform efforts at your school? Last year (2002-03) 
(mark all that apply) 
Whole-school qo 


Reading/language arts 


o 


Mathematics 
Science 

Social studies 
Arts 

Other (specify) 


mo 2 9 
EO fe O ee Oo 


ga 


16. Which of the following factors contribute to selection of a reform at your Last vear (2002-63 


school? 
(mark all that apply) 
a. Cost of reform o 
b. Theoretical or research foundation 


c. Successful implementation at other schools 
d. “Fit” with school needs 

e. State or district mandate 

f. Compatibility with assessment tools 


g. Best published results 


h. Don’t know 
i. Other (specify) 


gQoOaogo0nudcUuauddvundnd 
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IV. Faculty Role in Reform 


17. Did the faculty formally vote to adopt the current reform in your school? Last year (2002-03) 
(mark one response) 

a. Yes o 

No o 

c. N/A No adopted reform o 


18. Were you employed at this school when faculty voted formally to adopt the 


‘ Last year (2002-03) 
current reform? 


(mark one response) 


a. Yes o 

No o 

c. N/A No formal vote or adopted reform o 
19. Do you willingly participate in reform-related activities at your school? Last year (2002-03) 
(mark one response) 

a. Yes o 

b. No o 


V. School Performance Goals 


20. Does your school have performance goals for students at the end of each 


Last year (2002-03) 
school year? 


(mark all that apply) 


a. Yes, by grade level o 
b. Yes, by content area o 
c. Other (specify) o 
d. No Oo 


21. Does your school have intermediate student performance goals within each 


q Last year (2002-03 
school year to gauge student progress? 


(mark all that apply) 


a. Yes, by grade level o 
b. Yes, by content area Oo 
c. Other (specify) o 
d. No o 
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22. Which academic subjects are covered by goals or benchmarks at your 


. Last year (2002-03) 
school for student achievement? 


(mark all that apply) 


a. Reading/language arts 0 
b. Mathematics o 
c. Science a) 
d. Social Studies o 
e. Arts 0 
f. Other (specify) 0 


23. Does your school have a formal test preparation program to prepare 


Last year (2002-03) 
students for mandated state tests? eee 


(mark one response) 
a. Yes o 
b. No (Skip to question 26) qo 


24. If so, which grade levels participate in the test preparation program? (mark all that apply) 


Hoa @ @ @ @ B&B @ OD) GQ  Gmarkal that apply) 


25. Ifso, how much time do students spend in the test preparation program Lest vear (2002-03 


each year? 
(mark one response) 
a. 2 days or less Oo 
b. Between 2 and 5 days 


a 


Between 5 and 10 days 
More than 10 days 


QOodo 


Pp. 


26. Which one of these external entities is primarily responsible for supporting 


: Last year (2002-03) 
reform efforts at your school? (choose only one) 


(mark one response) 


a. University 0 
b. Regional Education Laboratory o 
c. State agency o 
d. School district o 
e. Reform program developer i) 

Teachers from other schools o 
g. Non-affiliated consultants 0 
h. Other (specify) o 
i. None, our school does not use external support 0 
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27. What kind of assistance does this primary external entity provide to support 


Last year (2002-03) 
reform efforts at your school? 


(mark all that apply) 


a. Onsite consulting 0 
b. Professional development o 
c. Networking opportunities 0 
d. Written materials for students o 
e. Written materials for teachers o 
f. Software or technological assistance o 
g. Other (specify) 0 
h. None, our school does not use external support 0 


28. What type of additional support has been available through your district 


. Last year (2002-03) 
for school reform efforts? 


(mark all that apply) 
Administering a needs assessment ) 


oe 


Providing additional school staff to support school reform 
Selecting a school reform model 

Writing grants to support school reform 

Providing professional development for school reform 
Release time for teachers 

Other (specify) 
None, the district does not supply additional support 


> +» @ 2 o 
Ouaoagauoaeuduo 


29. What type of additional support has been available through your state for 


Last vear (2002-03) 
school reform efforts? 
(mark all that apply) 


Administering a needs assessment i) 


Ss 


Selecting a school reform model 

Writing grants to support school reform 

Providing professional development for school reform 
Release time for teachers 

Other (specify) 
g. None, the state does not supply additional support 


me 20 


g0oago0Uunu6dcnd 
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VI. Instructional Practice and Professional Development 


30. What teacher enhancement opportunities are available to teachers in your 


school? 


o 


moe 2 © 


ge 


Mentoring new teachers 

Coaching other teachers 

Making management decisions 

Making decisions related to curriculum development 
Participating on grade-level teams 

Participating on content area teams across grades 
Other (specify) 


31. Are teachers scheduled to have a common planning period during the 


day? 


oreo 


Yes, by grade level 
Yes, by content area 
Other (specify) 


No, no common planning time is scheduled 


32. Do any of the following grouping patterns occur in your school (for some 
or all of the students in grades participating in reform)? 


oa aS 


Students remain with one teacher for most subjects 
Students are divided into groups such as “houses” or “families” 
Student groups remain two or more years with the same teacher 


Interdisciplinary teaching (e.g., two or more teachers with different 
academic specializations collaborate to teach an interdisciplinary 
program to the same group of students) 


Paired or team teaching (e.g., two teachers are jointly responsible for 
teaching a single group of students) 


None of the above 


Last vear (2002-03) 


(mark all that apply) 
o 


ogoaaqa0qag0.udouun 


Last year (2002-03) 


(mark all that apply) 
o 


goad 


Last vear (2002-03 
(mark all that apply) 
o 
qo 
o 
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33. Ifyou are a secondary school teacher, what kind of student groups exist to 


. 7 ;: : P Last year (2002-03) 
achieve particular instructional goals? 


(mark all that apply) 
a. Ability grouping o 
b. Academic or technical academies qo 
c. School-to-career paths o 
d. No-such grouping exists qo 
e. Other (specify) o 
f. Not applicable qo 
34. Who organizes the scope and sequence of the curriculum at your school? Last year (2002-03) 
(mark all that apply) 
a. State o 
b. District go 
c. School o 
d. Individual teachers o 
35. How is professional development organized at your school? Last year (2002-03) 
(mark all that apply) 
Teachers select activities sponsored by the district or other organizations o 
b. The district identifies common professional development themes for all o 
teachers 
c. The school reform plan includes professional development activities for o 
all teachers 
d. All teachers participate in the same professional development activities qo 
e. Each teacher has an individual professional development plan o 


36. Which kinds of professional development opportunities are available to 


Last year (2002-03) 
teachers at your school? 


(mark all that apply) 


a. Reading/language arts instruction o 
Mathematics instruction qo 
c. Instructional strategies for low-achieving, limited-English-proficient, o 


special education, and/or migrant students 


d. Ensuring that curriculum and instruction are consistent with state and/or o 
district content standards 


e. Ensuring that curriculum and instruction are consistent with state and/or o 
district assessments 


f. Implementation of a school reform model 


Monitoring individual students’ progress toward learning goals 


> 9p 


Interpreting reports of student achievement data 
Other (specify) 


googda 
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37. In what format or setting is professional development offered to teachers 


Last year (2002-03) 
at your school? 


(mark all that apply) 


a. Seminars o 
b. Workshops qo 
c. Study groups o 
d. Make and take qo 
e. Regional and national conferences qo 
f. Courses for college credit qo 
g. Other (specify) o 
h. None, our school does not provide formal professional development i) 


38. How is time allocated to enable teachers to participate in formal 


P eas Last year (2002-03) 
professional development opportunities? 


(mark all that apply) 


a. Pupil-free days o 
b. Release time 

c. Faculty meeting time o 
d. Personal time (evenings and weekends) o 
e. Holidays (including summer vacation) qo 
f. Our school does not provide formal professional development qo 


39. Does your school provide the equivalent of over 10 days of professional 


Last year (2002-03 
development per year? 


(mark one response) 


a. Yes o 
b. No og 


VII. Role of Parents 


40. By what means does your school formally communicate with parents? Last year (2002-03) 
(mark all that apply) 
a. Designated parent coordinator qo 
b. Regular newsletter 
c. Telephone calls 
d. Email 
e. Website 


f. In the language spoken at home (other than English) 
g. Other (specify) 


h. No formal communication with parents 


gQoagaaqgaagoada 
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41. Estimate the percentage of parents who support their children’s learning 
in the following ways (write percentage in space at right): Last year (2002-03) 


i. Monitor attendance from home 
j. Help students with homework 

k. Volunteer to assist in classrooms 
1. Other (specify) 


42. Estimate the percentage of parents who are engaged with the school in the 
following ways (write percentage in space at right): Last year (2002-03) 


m. Attend parent-teacher conferences 

n. Demand frequent reports on their children’s progress 

Actively volunteer in the school 

Observe classroom activities 

Actively participate in a formal parent organization (PTA/PTO) 
Other (specify) 


A) he 
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VIII. Reform Designs in Use 


Are any of the following design(s) currently in use at your school? (mark all that apply) 


Entire-School Reading/Language Arts 
O Accelerated Schools © Breakthrough to Literacy 
O America’s Choice Carbo Reading Styles Program 
O ATLAS Communities O CELL/ExLL 
© Audrey Cohen College: Purpose-Centered Education O CORE 
O Center for Effective Schools Early Intervention in Reading 
©) Child Development Project © Exemplary Center for Reading Instruction 
O} Coalition of Essential Schools 1 First Steps 
O Community for Learning Junior Great Books 
O Co-nect Literacy Collaborative 
O Core Knowledge O National Writing Project 
) Different Ways of Knowing O Reading Recovery 
1 Direct Instruction O Other reading/language arts design 
) Edison Schools 
O Expeditionary Learning Outward Bound Mathematics 
©) High Schools That Work O Connected Mathematics Project 
©) High/Scope Primary Grades Approach to Educ. © Core Plus Mathematics Project 
© Integrated Thematic Instruction © Growing with Mathematics 
O MicroSociety® Interactive Mathematics Program 
O Modern Red Schoolhouse O MATH Connections® 
O Montessori OU. of Chicago School Mathematics Project 
© Onward to Excellence © Other mathematics design 
O Paideia 
 QuEST Science 
O Roots & Wings 1 Developmental Approaches to Science, Health and 
Technology 
School Development Program (Comer) © Foundational Approaches in Science Teaching 
Success for All O GALAXY Classroom Science 
©) Talent Development HS with Career Academies © Iowa Chautauqua Program 
©) Talent Development Middle School © Other science design 
O) The Learning Network 
O) Turning Points Other 
© Urban Learning Centers O ACCESS 
O Ventures Initiative and Focus® System 0 Basic Skill Builders 
O Other whole-school reform 1 COMP: Creating Conditions for Learning 
O Feuerstein’s Instrumental Enrichment 
O HOSTS 
O HOTS 
1 Lightspan Achieve Now 
Positive Action 
Responsive Classroom 
O Success-in-the-Making 
O Other - not listed on this page 


Thanks for your participation! As soon as we receive your questionnaire, remuneration for you or 
your school will be mailed immediately. 
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Teacher Survey Year 2 


Longitudinal Assessment of 


Comprehensive School Reform 


TEACHER SURVEY 
2004-2005 School Year 


According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information 
unless such collection displays a valid OMB control number. The valid OMB control number for this information collection 
is 1875-0222. The time required to complete this information collection is estimated to average 20 minutes per 
response, including the time to review instructions, search existing data resources, gather the data needed, and 
complete and review the information collection. If you have any comments concerning the accuracy of the time 
estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 
20202-4651. If you have comments or concerns about the contents of this questionnaire, write directly to: 


WestEd, attn: John Flaherty, Project Coordinator, 4665 Lampson Avenue, Los Alamitos, CA 90720 
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Dear Teacher, 


WHO IS CONDUCTING THIS SURVEY? 

WestEd and Cosmos Corporation, educational research organizations under contract to the U.S. 
Department of Education, request your participation in this survey. WestEd and the Department 
of Education are conducting this survey by the authority of Section 1607 of the No Child Left 
Behind Act of 2001 (P.L. 107-110). 


WHAT IS THE PURPOSE OF THIS SURVEY? 

This survey is designed to understand how comprehensive reform at your school changes the 
ways that you and your staff approach teaching and learning. This survey focuses on changes 
that can affect every facet of school operations including classrooms, professional development, 
school-wide operations (e.g. scheduling) and management, and administration of the school 
building. 


HOW WILL THE RESULTS BE USED? 

The data from this survey will be used by the U.S. Department of Education and Congress to 
evaluate the implementation and effectiveness of federal school reform legislation. Data will be 
reported only in statistical summaries; your individual responses will be kept confidential. 


WHY SHOULD YOU PARTICIPATE IN THIS SURVEY? 

We are conducting this survey with only a sample of schools. Therefore, the value of your 
individual contribution is greatly increased because it represents many other schools. We 
encourage you to participate in this voluntary survey. 


WHERE SHOULD YOU MAIL YOUR COMPLETED QUESTIONNAIRE? 

Please return your completed questionnaire in the enclosed envelope. If you do 

not have the return envelope, you can mail the survey to the address below, or contact the study 
team via email or using our toll-free number provided below. 


Longitudinal Assessment of Comprehensive School Reform 
Attn: Susan Cragle 

55 Hanover Lane 

Chico, CA 95973 

1-866-880-2773 

csr@duerrevaluation.com 


THANK YOU FOR YOUR COOPERATION IN THIS IMPORTANT EFFORT! 
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Professional Development 


The questions in this section refer to all activities intended to help teachers develop and improve their 
content knowledge and classroom instruction. Examples of professional development activities include 


mentoring programs and coaching in addition to more traditional activities such as internships, workshops, 
conferences, institutes, and college courses. These may be conducted within the school or outside the 
school setting. 


1. During the last school year (2003-04, including summer 2004), how many of the following 
types of professional development activities did you participate in? (Circle one number in each 


row.) 
11 or 
ges - 1-2 3-5 6-10 ° 
Type of Professional Development Activity None ee fags Base more 
activities activities activities aN 
activities 
a. Conferences lasting two days or longer 1 2 3 4 5 
b. Institutes (i.e., an intensive course of 
instruction on a particular topic or set of l 2 3 4 5 
topics) lasting two days or longer 
c. A series of connected workshops lasting two \ > 3 4 5 
days or longer 
d. Workshops lasting one day or less l 2 3 4 5 
e. Courses for college credit 1 2 3 4 5 
f. Internships l 2 3 4 5 
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2. During the last school year (2003-04, including summer 2004), how frequently did you 
engage in each of the following professional development activities? (Circle one number in each 


row.) 
Once ora Once or Once or Daily or 
Type of Professional Development Activity few times _twicea twice a almost 
Never : 
a year month week daily 
a. Planned lessons or courses with other 1 2 3 4 5 


teachers 


b. Consulted with other teachers about 
individual students (e.g., discussing specific 1 2 3 4 5 
students and arranging appropriate help) 


c. Exchanged feedback with other teachers 
based on classroom observations (e.g., a 
teachers’ observation of your class, your 1 2 3 4 5 
observation of another teachers’ class, or 
observation of a class via video) 


d. Exchanged feedback with other teachers 


l 2 3 4 5 
based on student work 
e. Acted as a formal or informal coach or 
S 1 2 3 4 5 
mentor to other teachers or staff 
f. Received formal or informal coaching or 1 > 3 4 5 


mentoring from other teachers or staff 


g. Was observed/evaluated by the school 
principal or other staff (e.g., department 1 2 3 4 5 
chair, master teacher) 


h. Participated in a learning community (e.g., 


teacher collaborative, network, or study 1 2 3 4 5 
group) 

i. Participated in a district or school committee 
focused on curriculum, instruction, or student 1 2 3 4 = 
assessment 

j. Visited other schools to observe classroom 1 ; 3 4 5 


teaching and learning 
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3. During the last school year (2003-04, including summer 2004), about how many hours of 
professional development did you receive in each of the following areas? (Circle one number in 
each row.) 


More 
er ee -5 -2 25 - . 
Area of Professional Development None — 240. Ah than 80 
hours hours hours hours 
hours 
a. Instructional strategies for teaching 
: 1 2 3 4 3 6 
reading/language arts/English 
b. In-depth study of topics in 
hee ; : l 2 3 5 6 
reading/language arts/English 
C, Instructional strategies for teaching 1 2 3 4 5 6 
mathematics 
d. In-depth study of topics in mathematics 1 2 3 4 5 6 
e. Instructional strategies or in-depth study 
of topics in other academic subject (e.g., 
; x 5 = 1 Ds 3 4 5 6 
science, social studies, foreign language, 
etc.) 
f. Instructional strategies for students with \ > 3 4 5 6 
limited English proficiency (LEP) “ 
g. Instructional strategies for students with 
Nae : ; 1 Ps 3 4 5 6 
Individualized education programs (IEPs) 
h. Preparing students to take the annual state , > 3 4 5 6 
assessment 
1. Analyzing and interpreting student 1 > 3 4 5 6 
achievement data 
j. Classroom and behavior management 1 2 3 4 5 6 
k. Use of technology to improve classroom \ > 3 4 5 6 
instruction 
l. Use of appropriate assessment 1 > 3 4 5 6 
accommodations 
m. Family/community involvement 1 2 3 4 5 6 
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4. Thinking of all of the different professional development activities 
that you participated in during the last school year (2003-04, 
including summer 2004) and reported in this section, approximately 
how many total hours did you spend in professional development? 
(Write in the number of hours.) hours 


5. Indicate the number of hours allocated for professional development from each of the 
following sources during the last school year (2003-04, including summer 2004). Include only 
workshops, coursework, and conferences sponsored by your school, district, or state. (Circle 
one number in each row.) 


Source of Professional Development 1-5 6 - 24 25-40 41-80 More 
: None than 80 

Time hours hours hours hours ; 

hours 
a. Pupil-free days l 2 3 4 5 6 
b. Release time l 2 3 4 5 6 
c. Faculty meeting time 1 2 3 4 5 6 
d. Personal time (evenings and weekends) 1 2 3 4 5 6 
e. Holidays (including summer vacation) 1 2 3 4 5 6 


Professional Community 


6. Are teachers at your school scheduled to have a common planning 


° ° ‘a (mark all that apply) 
period during the day? 


a. Yes, by grade level O 
b. Yes, by content area O 
c. Other (specify) O 
d. No, no common planning time is scheduled O 
7. What professional opportunities are available to teachers in your ae rene 
school? 
a. Mentoring new teachers O 
b. Coaching other teachers O 
c. Making management decisions O 
d. Making decisions related to curriculum development 0 
e. Participating on grade-level teams a 
f. Participating on content area teams across grades O 
g. Other (specify) O 
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School Reform 
The following questions refer to the most important reform strategy or strategies in your school. Such 
reform strategies are those intended to improve school operations and student outcomes. 


8. To what extent do you believe reform at your school: (Circle one number in each row.) 
Toa 


be To some fairl Toa 
Not at all small : ied great 
extent large 
extent extent 
extent 
a. helps to improve student learning 1 2 3 4 5 
b. helps you to teach more effectively 1 2 3 4 5 
c. detracts from more important efforts 1 2 3 4 5 
d. improves communication among teachers 1 2 3 4 5 
e. helps to improve student behavior 1 2 3 4 5 


Parent Involvement 


9. Thinking about the parents of the students in your classroom(s), 
what percentage of those parents support their children’s learning 
in the following ways: (write percentage in space at right) 


a. Monitor attendance from home % 
b. Help students with homework % 
c. Volunteer to assist in classrooms % 
d. Other (specify) % 


10. Thinking about the parents of the students in your classroom(s), 
what percentage of those parents are engaged with the school in the 
following ways: (write percentage in space at right) 


a. Attend parent-teacher conferences % 
b. Demand frequent reports on their children’s progress % 
c. Actively volunteer in the school % 
d. Observe classroom activities % 
e. Actively participate in a formal parent organization (PTA/PTO) % 
f. Other (specify) % 
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Reform Strategies 


Schools often use single or multiple reform strategies designed locally or by external program developers. Some 
strategies are externally designed, for example Success for All or Co-nect. Other strategies are local and may not have a 
formal name. On this page we are collecting specific information about each strategy in use at your school. Begin by 


writing the name of the most important reform strategy at the top of the first column and then answer each question 
considering that strategy only. Then, using the next column, repeat the process again for each additional strategy at 
your school. If your school is using a single strategy, complete the first column only. If your school has no reform 


strategy, write “none” in the first column. 


Reform Strategy Name PIII IFIIFD D> 


Description of Reform Strategy 


Does the strategy target any curricular areas for 
improvement? 
If yes, indicate curricular areas targeted: 
(Circle all that apply.) 


Does the strategy target school organization and 
management for improvement? 

Does the strategy target community and parent 
involvement? 

Do you have evidence that this strategy improves 
student achievement? 

Does the evidence include comparison groups? 

Has the strategy been demonstrated to improve student 
achievement at other schools? 


Status of Reform Strategy 


Are all grades included in the strategy? 

Is this strategy being phased-in? 

Is the strategy accompanied by benchmarks for 
implementation? 

Did your district mandate this strategy? 

Will sanctions result from failure to implement this 
strategy? 


Support Provided through Reform Strategy 


Was this strategy developed externally? 

Does the strategy include specific curricular materials? 
Do guidelines for classroom practice accompany the 
strategy? 

How many hours of training on the strategy did you 
receive from the program developer? 

How many hours of training on the strategy did you 
receive from the district? 

How many hours of training on the strategy did you 
receive from the state? 

How many of these total training hours (from the 
developer, district and state) occurred at your school? 
Is ongoing support for the strategy available through an 
onsite facilitator? 
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(circle one) 


Yes No 


Language Arts 
Mathematics 
Social Studies 
Science 
Arts 
Other 


(circle one) 


Yes No 
Language Arts 
Mathematics 
Social Studies 
Science 
Arts 
Other 


Yes 
Yes 


Yes 
Yes 
Yes 


(circle one) 


Yes No 


Language Arts 
Mathematics 
Social Studies 
Science 
Arts 
Other 


Yes 
Yes 


Yes 
Yes 
Yes 


hours 
hours 


hours 


total 
hrs. 


Yes No 


(circle one) 


Yes No 


Language Arts 
Mathematics 
Social Studies 
Science 
Arts 
Other 


Yes 
Yes 


Yes 
Yes 
Yes 


hours 
hours 


hours 


total 
hrs. 


Yes No 
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Background 


11. How many years have you been a teacher? years 
12. How many years have you been a teacher at this school? years 
13. What type of credential(s) do you hold? (mark one response) 
a. Professional 0 
b. Preliminary 0 
c. Emergency O 
d. Substitute O 
e. Other Oo 
14. Which best describes your MAIN teaching assignment? (mark one response) 
a. Self-contained classroom teacher O 
b. Specialist teacher (mark below your primary subject area assignment oO 
for this year). 
C1 English as a second language C1 Science 
C1 Fine Arts C1 Special Education 
CL] Language Arts CL] Social Studies, History, Government 
OO Mathematics oO Speech, Communication 
1 Physical Education (]_ Writing Specialist 
CL] Reading Specialist CO Other 


THANK YOU FOR PARTICIPATING! 


Please return your completed questionnaire in the enclosed envelope. If you do 
not have the return envelope, you can mail the survey to the address below, or contact the study 
team via email or using our toll-free number provided below. 


Longitudinal Assessment of Comprehensive School Reform 
Attn: Susan Cragle 

55 Hanover Lane 

Chico, CA 95973 

1-866-880-2773 

csr@duerrevaluation.com 
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Teacher Survey Year 5 


Longitudinal Assessment of 


Comprehensive School 
Reform 


TEACHER SURVEY 


2006-07 School Year 


According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless 
such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1875- 
0222. The time required to complete this information collection is estimated to average 40 minutes per response, including the 
time to review instructions, search existing data resources, gather the data needed, and complete and review the information 
collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, 
please write to: U.S. Department of Education, Washington, D.C. 20202-4651. If you have comments or concerns about the 
contents of this questionnaire, write directly to: 


WestEd, attn: John Flaherty, Project Coordinator, 4665 Lampson Avenue, Los Alamitos, CA 90720 
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Dear Teacher, 


WHO IS CONDUCTING THIS SURVEY? 

WestEd and Cosmos Corporation, educational research organizations under contract to the U.S. 
Department of Education, request your participation in this survey. WestEd and the Department 
of Education are conducting this survey by the authority of Section 1607 of the No Child Left 
Behind Act of 2001 (P.L. 107-110). 


WHAT IS THE PURPOSE OF THIS SURVEY? 

This survey is designed to understand how comprehensive reform at your school changes the 
ways that you approach teaching and learning. This survey focuses on changes that can affect 
every facet of school operations including grade-level planning, curriculum and instruction, data 
driven decision making, professional development and alignment with state and district reform 
efforts and priorities. 


HOW WILL THE RESULTS BE USED? 

The data from this survey will be used by the U.S. Department of Education and Congress to 
evaluate the implementation and effectiveness of federal school reform legislation. Responses to 
this data collection will be used only for statistical purposes. The reports prepared for this study 
will summarize findings across the sample and will not associate responses with a specific 
district or individual. We will not provide information that identifies you or your district to 
anyone outside the study team, except as may be required by law. 


WHY SHOULD YOU PARTICIPATE IN THIS SURVEY? 

We are conducting this survey with only a sample of schools. Therefore, the value of your 
individual contribution is greatly increased because it represents many other schools. We 
encourage you to participate in this voluntary survey. 


WHERE SHOULD YOU MAIL YOUR COMPLETED QUESTIONNAIRE? 

Please return your completed questionnaire in the enclosed envelope. If you do not have the 
return envelope, you can mail the survey to the address below, or contact the study team via 
email or using our toll-free number provided below. 


Longitudinal Assessment of Comprehensive School Reform 
Attn: Susan Cragle 

55 Hanover Lane 

Chico, CA 95973 

1-866-880-2773 


scragle@duerrevaluation.com 


THANK YOU FOR YOUR COOPERATION IN THIS IMPORTANT EFFORT! 
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Grade-Level and Content Planning 


hee 


On average during the current school year, how many hours per 
week have you spent with others in common planning time? (Write 
in the number of hours) hours per week 


2. During the last 12 months, how frequently have you met formally or informally with others 
at your school to conduct the following activities related to planning and teaching? (Circle one 
number in each row) 

Once ora Once or Once or Daily or 


few times _twicea twice a almost 
Never a year month week daily 
a. Participate in common planning period for 
1 D: 3 4 5 
teachers grouped by grade level 
b. Participate in common period for teachers 
1 2 3 4 5 
grouped by content area 
c. Receive training or guidance on the role or 1 > 3 4 5 


purpose of common planning periods 


d. Consult with other teachers about individual 
students (e.g., discussing specific students and 1 2 3 4 5 
arranging appropriate help) 


e. Exchange feedback with other teachers based 
on classroom observations (e.g., a teacher’s 
observation of your class, your observation of 1 2 3 4 5 
another teacher’s class, or observation of a 
class via video) 


f. Exchange feedback with other teachers based 


on student work d : ? i ° 

g. Actas a formal or informal coach or mentor to I > 3 A 5 
other teachers or staff 

h. Receive formal or informal coaching or 1 > 3 4 5 
mentoring from other teachers or staff 

i. Be observed/evaluated by the school principal 
or other staff (e.g., department chair, master 1 D, 3 4 5 
teacher) 

j. Participate in a learning community (e.g., 
teacher collaborative, network, or study 1 2 3 4 5 
group) 

k. Participate in a district or school committee 
focused on curriculum, instruction, or student 1 2 3 4 5 
assessment 

1. Visit other schools to observe classroom I > 3 4 5 


teaching and learning 
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3. Who participates in these planning activities? (Circle one number in each row) 
Once ora Once or Once or Daily or 


few times _twicea twice a almost 
Never a year month week daily 
a. Classroom teachers 1 2 3 4 5 
b. Teachers on special assignment 1 2 3 4 5 
c. Curriculum coaches 1 2 3 4 5) 
d. School administrators 1 2 3 4 5 
e. Teachers from other schools 1 2 3 4 5 
f. District officials 1 2 3 4 5 
g. State officials 1 2 3 4 3) 
h. Reform program developers 1 2 3 4 5 
i. Other (specify) 1 2 3 4 3) 


Coordinated Curriculum and Instruction 


4. How much does each of the following influence what is taught in the classroom? (Circle one 
number in each row) 


Strong Somewhat Little or Somewhat Strong 
negative negative no positive positive 
influence influence influence influence influence 

a. State curriculum frameworks or content 
1 2 3 4 5 
standards 
b. District curriculum frameworks or content 1 2 3 4 5 
c. Textbook/instructional materials 1 2 3 4 S) 
d. State testing requirements 1 2 3 4 5 
e. District testing requirements 1 2 3 4 5) 
f. Students’ special needs or individual I > 3 4 5 
instructional plans 
g. Parent or community preferences 1 2 3 4 3) 
h. Professional development experiences 1 2 3 4 5 
i. Diagnostic or classroom assessment results 1 2 3 4 5 
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5. Which entities — state, district or school officials — are 
involved in decision-making for each of the following? (nark ail that apph 


State District | School pons 

know 
a. Selecting textbooks and other instructional materials LJ fl L] L] 
b. Selecting content, topics, skills to be taught fa al [i=] LI 
c. Selecting teaching techniques | [al EI [eal 
d. Creating student ability groups for instruction CL Lal | L] 
e. Allocating instructional time for each academic subject il El L L 
f. Pease hit ae = service professional development gO Oo Og O 
g. Hiring new full-time teachers CL] (ea (el L 
h. Planning school budgets LJ L] L] CL] 
i. Determining professional and teaching assignments O CL il LC 
j. Establishing the school curriculum LJ L] L L 
k. Other (specify) LJ E) El LI 


6. Have you received any of the following designed to align curriculum 
and instruction with state and/or district content standards? (hark albthatapph 


Detailed standards-based curriculum guides 
Detailed pacing schedules 


c. Classroom-embedded assessments with a standard scoring rubric to monitor 
progress toward district or state standards 


d. Curriculum map aligning the required curricula and instructional programs to 
standards 


e. Curriculum map aligning the required curricula and instructional programs to 
assessments 


f. Model lesson plans based on the standards 


g. Analytic reports summarizing student achievement data to identify specific 
strengths and weaknesses related to the attainment of standards 


h. Information on listservs related to standards 


i. Professional development designed to ensure that curriculum and instruction are 
consistent with state and/or district content standards 


j. Professional development designed to ensure that curriculum and instruction are 
consistent with state and/or district assessments 


k. Other (specify) 


ht Ff Ffiet it 


Data Driven Decision Making 


7. Do you receive reports or a summary of assessment results from 


state or district tests? (mark one response) 
a. Yes L 
b. No L 
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8. Do you receive any of the following to support the use of assessment 
results? (mark all that apply) 


a. Test results disaggregated by classroom [mm 
b. Test results disaggregated for special populations of students LO 
c. Test results disaggregated by grade ie 
d. Test results disaggregated by sub-test | 
e. Item-by-item review of test results im 
f. School reports showing trends over three years or more | 
g. Training and consultation for school staff to allow school-based analysis of O 

student achievement data 

9. Did you receive state or district assessment results summarized by 

any of the following categories of students? nace all mataegh 
a. Race or ethnicity O 
b. Gender L] 
c. Title I participation im 
d. Migrant status L] 
e. Poverty status LJ 
f. Limited English Proficiency (LEP) status fe] 
g. Students Individualized Education Programs (IEPs) O 


10. Who at your school receives such reports or summaries of 


assessment results? (mark all that apph 
Principals/administrators 


Teachers 
Parents 
Students 
Don’t know 


on 9 5 P 
ff 


11. Have you received any assistance or training on how to interpret or 
analyze state or district assessment results? (mark all that apph 


Yes, from the state 
Yes, from the district 


Yes, from another external entity 


a9 5 pS 
OO 


No assistance or training on interpreting or analyzing assessment results 
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12. Which data sources are available to you and your school for student 


assessment or planning purposes? iidticail hat aeph 
a. “Raw” student performance data on state or district tests 


Reports summarizing student performance on state or district tests 


© 


Reports that analyze or interpret student performance on state or district tests 
based upon demographic characteristics (such as race or gender) or educational 
program (such as results for students with special needs) 


Reviews or evaluations of program implementation by external entity 
Diagnostic tests to measure student readiness 
Classroom observations by peers or administrators 


Reviews of student projects or portfolios 


= 0 -m © & 


Surveys of students, parents, teachers or staff regarding the school climate and 
satisfaction 


i. Research reports from external resources on the efficacy of improvement 
strategies 


1 ny = 6hUuL 


j. Research conducted at the school (e.g., teacher or administrator action research 
projects; staff analysis of school data) on the efficacy of current improvement 
strategies 


k. Tests of student knowledge or skills at the end of instructional units 


l. Tests of student knowledge and skills at regularly scheduled times (e.g., gO 
quarterly or every six weeks) 


(mark one response per row) 
13. To what extent have state or district assessment results 


been used for the following purposes? Little 
pee Moderat Great Don’t 
e use use know 
use 
a. Measuring student progress toward meeting state or district 
standards or benchmarks O O 0 O 
b. Identifying students for intervention or support services LJ L] LO LO 
c. Placement of students into remedial or advanced courses LJ | fe L 
Identifying areas where more professional development is 
needed a a CM OH 
e. Assessing teacher performance CL] L al L 
f. Helping teachers align their instruction to the school curriculum LJ LC] LC] O 
g. Establishing priorities for future instruction (e.g., more oO gO gO gO 
emphasis on subjects or topics where students scored low) 
h. Revising the school improvement plan (SIP) LF O O O 
i. Evaluating school initiatives/programs CL L] L L 
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School Improvement Plans 


14. How formal is the school improvement plan at your school? 
(mark all that apply) 


a. Comprehensive written plan | 
b. Outline of a plan im 
c. Written mission statement only LJ 
d. No formal plan but teachers generally share the same ideas L] 
e. Multiple plans im 
f. No plan (Skip to question 19) L] 


15. Does your school improvement plan cover the following? 


(mark all that apply) 
Measurable goals or objectives 


Se 


Mechanism for periodic evaluation of goals 
Curriculum and instruction content 

Student assessment rubrics 

Classroom management guidelines 
Professional development activities 


Parental involvement plan 


= 9m -» © 2 © 


Framework for participation in school management 


i. Integration of new technology 


OBO ORO eo 


j. Other (specify) 


16. What factors influenced the content of your formal school 


improvement plan? (mark all that apply) 
State or district content standards 


Se 


State or district performance standards 


Needs identified through a school needs assessment 


a 12 


School performance standards 

Specifications of adopted/adapted reform design 
Assigned by district/state 

Other (specify) 


Don’t know 


Om Oe Oe 


= fm = Oo 
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17. Which entities — state, district or school officials — are 
involved in decision-making for each of the following? (mark ail that apph 


State District School Dan 
know 
Developing the school improvement plan LJ i L] L] 
Reexamining the school improvement plan im LF im LO 
c. Providing data to review for the school improvement plan (e.g., 
standardized test data) r r r r 
d. Provide training on developing and/or reexamining the school 
improvement plan O r r r 
e. Establishing goals and benchmarks for the school O O O O 


— 
~ 


How often does your school review or monitor progress toward the 


. 9 
school improvement plan’s goals? (mark one response) 
Never 


Once every two years 
Annually 

Two or three times a year 
Quarterly 
Monthly 

Other (specify) 


Don’t know 


Dp ee > fom o fom o Fe 
OOOO 


Professional Development 


The questions in this section refer to all activities intended to help teachers develop and improve their 
content knowledge and classroom instruction. Examples of professional development activities include 
mentoring programs and coaching in addition to more traditional activities such as internships, 
workshops, conferences, institutes, and college courses. These may be conducted within the school or 
outside the school setting. 


19. How is professional development organized at your school? Include 
only workshop, coursework, and conferences sponsored by your 
school, district, or state during the last 12 months. (mark all thai apply) 


Teachers select activities sponsored by the district or other organizations 
The district identifies common professional development themes for all teachers 


c. The school reform plan includes professional development activities for all 
teachers 


d. All teachers participate in the same professional development activities 


Iti 


e. Each teacher has an individual professional development plan 
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20. During the last 12 months, how many of the following types of professional development 
activities did you participate in? (Circle one number in each row) 


1-2 a5 6-10 roe 
Type of Professional Development Activity None asvites:  Beusiee: wActwines more 
activities 
Conferences lasting two days or longer 1 2 3 4 3 


Institutes (i.e., an intensive course of 
instruction on a particular topic or set of 1 2 3 4 5 
topics) lasting two days or longer 


c. A series of connected workshops lasting two 


days or longer : : : 4 > 
d. Workshops lasting one day or less 1 2 3 4 5 
e. Courses for college credit 1 2 3 4 5 
Internships 1 2 3 4 5 


21. 
Indicate the number of hours allocated for teacher professional development from each of 
the following sources during the last 12 months. Include only workshops, coursework, and 
conferences sponsored by your school, district, or state. (Circle one number in each row) 
Mor 
Source of Professional Development Time None nase ee oe paki tan 80 
ours 
a. Pupil-free days 1 2 3 4 5 6 
b. Release time 1 2 3 4 5 6 
c. Faculty meeting time 1 2 3 4 5 6 
d. Personal time (evenings and weekends) 1 2 3 4 5 6 
e. Holidays (including summer vacation) 1 D) 3 4 5 6 
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22. 


During the last 12 months, about how many hours of professional development did you 
receive in each of the following areas? (Circle one number in each row) 


More 
: 1-5 6-24 25-40 41-80 
Area of Professional Development None Rous haus ous Baie than 80 


hours 

a. Instructional strategies for teaching 1 2 3 4 5 6 
Reading/Language Arts/English 

b. In-depth study of topics in Reading/Language 1 > 3 4 5 6 
Arts/English 

c. Instructional strategies for teaching 1 2 3 4 5 6 
Mathematics 

d. In-depth study of topics in Mathematics 1 2 3 4 5 6 


e. Instructional strategies or in-depth study of 
topics in other academic subject (e.g., Science, 1 2 3 4 5 6 
Social Studies, Foreign language, etc.) 


f. Instructional strategies for students with 


Limited English Proficiency (LEP) : 2 2 i : : 
g. Instructional strategies for students with 1 2 3 4 5 6 
Individualized Education Programs (IEPs) 
h. Preparing students to take the annual state 1 3 3 4 5 6 
assessment 
1. Analyzing and interpreting student 1 2 3 4 5 5 
achievement data 
j. Classroom and behavior management 1 2 3 4 5 6 
k. Use of technology to improve classroom 1 > 3 4 5 ; 
instruction 
1. Use of appropriate assessment I > 3 4 5 6 
accommodations 
m. Family/community involvement 1 2 3 4 5 6 
n. Ensuring that curriculum and instruction are 
consistent with state and/or district content 1 2 ) 4 5 6 
standards 
o. Ensuring that curriculum and instruction are 
consistent with state and/or district 1 D) 3 4 5 6 
assessments 
Implementation of a school reform model 1 2 3 4 5 6 
Monitoring individual students’ progress 
; 1 2 3 4 5 6 
toward learning goals 
r. Interpreting reports of student achievement 1 3 3 4 5 6 


data 


OMB No. 1875-0222 Approval Expires 09/30/2007 


Appendix B 148 Teacher Survey Year 5 


Thinking of all of the different professional development activities 
that you participated in during the last 12 months, and reported in 


this section, approximately how many total hours were spent in 
professional development? (Write in the number of hours) hours 


Adoption of Scientifically Based Methods 


24. 


25. 


27. 


28. 


oO aS 


Is your school using a school reform strategy or any combination of 


school reform strategies? (mark one response) 
Yes a 
No L 
Not sure CL 


Please name the most important reform strategy or strategies in use at your school. 


In what year did your school begin implementing the most 
important reform model or strategy? year 


At which grade levels in your school is reform mainly focused? 
oa BD we fF & @ 8 @ LB Mark alt that apply) 


How is reform at your school designed? (mark one response) 
Totally designed at this school 


Adapted with modifications from external source 


Adapted selected parts from multiple external sources 


Oe Oe 


Adopted unmodified from external source 
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29. 


Indicate the primary designer for the reform efforts at your school: — (mark one response) 
Locally developed 


School district 


University 
Private developer or publisher 
Other (specify) 


on 9 5 P 
J fy 


30. 


Which categories best describe the focus of reform efforts at your (mark all that apply) 
school? 


Whole-school 
Reading/Language Arts 


Ss & 


c. Mathematics 

d. Science 

Social studies 
f. Arts 

g. Other (specify) 


2 


2 ee 


Who is responsible for selecting the school reform in which your (mark all that apply) 


school participates? 
School board 


District central office 


Se 


School administrators 
School improvement team 
Teachers 
Parents 

Other (specify) 


Don’t know 


OmOR Oe oO 


= 9 -» © 2 © 
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32. 


Which of the following factors contribute to the selection of (mark all that apply) 
reform at your school? 


Cost of reform 


Ss 


Theoretical or research foundation 
Successful implementation at other schools 
“Fit” with school needs 

State or district mandate 

Compatibility with assessment tools 

Best published results 

Don’t know 
Other (specify) 


Capacity and Support for Reform 


The following questions refer to the most important reform strategy or strategies in your school. Such 
reform strategies are those intended to improve school operations and student outcomes. 


= md » © 2 © 
lH ff ft i 


i) 


33. 


Which of these external entities is primarily responsible for (mark one response) 
supporting reform efforts at your school? 


University 


Regional Education Laboratory 
State agency 

School district 

Reform program developer 
Teachers from other schools 


Non-affiliated consultants 


Bee > foe o foo Fe 


State-designated assistance providers 
Other (specify) 


j. None, our school dos not use external support 


i 


OBO Oe Oe Oe 
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34. 


What kind of assistance does this primary external entity provide ee 
(mark all that apply) 
to support reform efforts at your school? 


Onsite consulting 


Ss 


Professional development 
Networking opportunities 

Written materials for students 
Written materials for teachers 
Software or technological assistance 
Other (specify) 


= 0m -m © 2 © 


None, our school does not use external support 
Don’t know 


“Ro foe 


j=\5 


35. 


Was there a vote at your school to adopt or implement the current = aie 
= - : ( pplv) 
reform at your school? If so, who participated in the voting? 


Administrators 


S & 


Teachers 


c. Students 


A. 


Parents 


a 


e. None, there was no vote on the reform 


36. 


Do you willingly participate in reform-related activities at your (mark one response) 


school? 
. Yes O 
b. No CL 


37. To what extent do you believe reform at your school: (Circle one number in each row) 


Toa 
hae To som fairl Tos 
Not at all small oe y great 
extent large 
extent extent 
extent 
a. Helps improve student learning 1 z 3 4 5 
b. Helps teachers to teach more effectively 1 2 3 4 2) 
c. Detracts from more important efforts 1 z 3 4 3) 
d. Improves communication among teachers 1 2 3 4 5 
e. Helps to improve student behavior 1 2 3 4 5 
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38. 
What factors limit teacher participation in reform at your school? (mark all that apply) 


No limitations — all teachers participate at this school 


Ss 


Our reform is subject-specific — only teachers in a specific subject participate 


c. Our funding limits the number of teachers who can participate 


a 


Our school reform is being phased in over time 


fit” 


e. Other reason (specify) 


39. To what extent do you disagree or agree with the following statements about the academic 
program at your school? (Circle one number in each row) 


Strongly Disaared To some Apies Strongly 
disagree extent agree 


a. At this school, we have a common 
understanding of the objectives we’re trying to 1 2B 3 4 5 
achieve with students 


b. Goals and priorities for the school are clear 1 2 3 4 5 
c. Most teachers at this school have values and 
; 3 Beale Cons 1 2 3 4 5) 
philosophies of education similar to my own 
d. Most of my colleagues share my beliefs and 
values about what the central mission of the 1 2 3 4 5 
school should be 
e. Most staff in this school share a focused 
ae é 1 D 3 4 5 
vision for student learning 
f. The principal communicates a clear academic I > 3 4 5 


vision for our school 


g. Teachers are committed to developing 
partnerships with parent(s)/guardian(s) for 1 2 3 4 3) 
student learning 


h. The school encourages parents to be involved 
in defining the school mission and goals 
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40. To what extent does your school have the resources 
needed to implement the major elements of the reform 


at your school? (mark one response per row) 


Not Somewhat Definitely 
available available available 
Design team materials to support instruction E) LJ L] 
b. Availability of design team materials to further implement the 
design in your school 
c. Professional development for teachers [a LJ LI 
d. Time for planning, collaboration, and development [eI CL] cal 
taff or consultants to mentor, advise, and provide ongoin 
ee : nie O O O 
f. Technology and connectivity LJ CL] L 
g. Funds i L] L 
h. Funding flexibility C | L] 
i. Support from staff L] LI a 


Background 


41. How many years have you been a teacher? years 


42. How many years have you been a teacher at this school? years 


43. 
What type of credential(s) do you hold? (mark all that apply) 


Professional 


S & 


Preliminary 
c. Emergency 
Substitute 
e. Other 


a 
| i) 
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44. 


Which best describes your MAIN teaching assignment? 


Self-contained classroom teacher 


s 


THANK YOU FOR PARTICIPATING! 


Specialist teacher (mark below your primary subject area assignment 


for this year) 


Fine Arts 


Mathematics 


Oe oO mo 


English as a second language 


English/Language Arts 


Physical Education 


Oo Reading Specialist 


Grade level (mark below your grade level assignment for this year) 


Om oO mo 


O 


(mark one response) 


O 


Science 
Special Education 


Social Studies, History, 
Government 


Speech, Communication 
Writing Specialist 
Other 


Ko @ @ 


5] (6) @) 8) 2) bo Dy) 2 (markall that appl) 


Please return your completed questionnaire in the enclosed envelope. If you do 
not have the return envelope, you can mail the survey to the address below, or contact the study 


team via email or using our toll-free number provided below. 


Longitudinal Assessment of Comprehensive School Reform 


Attn: Susan Cragle 

55 Hanover Lane 

Chico, CA 95973 
1-866-880-2773 
csr@duerrevaluation.com 
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Field Study Protocol 


CHECKLIST OF MATERIALS 
TO BE OBTAINED AND REVIEWED 
BEFORE THE 2ND SITE VISIT 


Instruments to Bring: 


1. A blank “Summary Measures of 11 CSR Components” - 2 copies (5 pages); 
2. The original site visit protocol, entitled Field-based Component Site Visit Inventory for 
LACIO (4 pages); 
3. The original Adjustment to Bodilly Scale (1 page); 
4a. A blank Classroom Observation Form (6 copies per visitor) (5 pages); 
*4b. A new addendum to the Site Visit Protocol (1 page); 
*5. A list of School-Level Loose Ends for Field Study Teams for 2004-05 (1 page); and 
*6. Data Collection Topics for District and State Officials for Field Study Teams, 2004-05 
(1 page) 


First Round Data Collection To be Reviewed: 


8. DO NOT REVIEW THE EARLIER (2003-2004) “Summary Measures of 11 
Components”; 
9. Long version of the case study report with Robert’s hand-written comments, if any; 
10. Short version of the case study report for each school (1 page each); 
11. The filled out instrument, if any, from the Telephone Inventory with the specific district 
and state officials associated with your site. 


Other: 
12. Honorarium Form- WestEd - 2 copies (1 page); and 


13. Background information about the 11 CSR components (9 pages); and 
14. Review any materials received from the schools prior to the upcoming visit. 


*Items to be discussed in phone prep (group) meeting prior to the site visit 
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Guidance for Completing the LACIO' 
Summary Measures of 11 Components 


Please complete the attached Summary of Measure of 11 Components Form during 
the 2003 field-based site visits. This form should be completed for both the intervention 
and the comparison sites. 


* Many items on this form require teams to make judgment calls. Keep in mind 
that this form is intended to quantify, as best possible, the level of implementation 
and reform at 30 different schools, using widely varying reform methods, and 
implementing these methods in widely varying ways. Use your best judgment and 
feel free to write clarifying comments as necessary. A good technique is for each 
team member to complete the form separately, share the results, and eventually 
arrive at a consensus. 


* It also should be noted that several items ask for judgments on activities or 
events that may be occurring over time. For example, Item 8.1 asks for the level 
of developer support over time, and 11.2 asks for student achievement increases 


since CSR funding. Site visit teams should respond to these measures for the 
current period (i.e., is developer support “high” at the time of the first visit). 


* This is a summary measure of “Comprehensive School Reform,” and is not 
limited to the research-based method (components | and 11). When completing 
this form, site visitors should respond to components 2 through 10 with this broader 
vision in mind. For example, if the school participates in professional development 
not related to the adopted research-based method, that professional development 
should still be considered when completing component 3. The team should also 
make notations indicating whether this additional professional development is 
aligned to the comprehensive reform design or plan. 


Site visit team members will be notified if it becomes necessary to amend these instructions. 
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Summary Measures of 11 Components 


Component Measure Score* 
1. Research-Based Method or Strategy 
1.1 Implementation score (adjusted Bodily Scale): 


1.2 Percentage of classrooms implementing (that should have been using 
the method): 


1.3 Fidelity rating by developer or consultant (high, medium, low, defined 
as follows): 


high: developer/consultant considers school to be among the best 
seen 
medium: developer/consultant considers school to be using method in medium 
acceptable manner 
low: developer/consultant has major complaints about school’s use low 


of method 


Total Possible Score for Component 1 9 
2. Comprehensive Design 
2.1 Existence of written design or plan (name it and give its date): 


Name: 
Date: 


2.2 Contents of plan (yes/no to each item): 
2.2.1 Inclusion of needs assessment or other performance data 
2.2.2 Reference to specific financial resources 
2.2.3 Indication of strategic use of financial resources 
2.2.4 Statement of quantitative performance goals 
2.2.5 Discussion of specific curricula 
2.2.6 Discussion of assessment tools/data-driven instruction 
2.2.7 Discussion of a professional development plan/strategy 
2.2.8 Suggestions for organizational or structural changes within the 
school 
2.2.9 Discussion of reform method sustainability. 


en ee 


2.3 Breadth of plan in covering all school operations (including, implicitly, 
all other CSR components) (high, medium, low, defined as follow): 


high: covers all CSR components (whether implicitly or explicitly) high 3 
medium: covers four or five components, but not all medium 2 
low: covers one to three components only (also name them) low 1 
Total Possible Score for Component 2 13 
* yes=1 and no=0 
(Continued) 
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Component Measure Score* 
3. Professional Development 
3.1 Strong content focus: 
3.2 Use of non-traditional teaching strategies: 


3.3 Range of PD hours required or taken by average teacher per year: 


3.4 Aligned with the student needs/data-driven instruction: 


3.5 Evidence of collective participation of groups of teachers within the 
same school: 


3.6 Evidence of some PD taking place in the teacher’s classroom: 


3.7 Explicit guidance to align PD with standards, curriculum, or assessment 
tools: 


4. Measurable Goals and Benchmarks 


4.1 Number of academic subjects covered: 
(Math, science, social science, English, language arts, social studies, 
vocational education, etc.) 


4.2 Number of grades covered and total no. of grades in the school: 


Total Possible Score for Component 4 4 
5. Support within the School 
5.1 Existence of formal faculty votes on reform or research-based method: 


5.2 Formal faculty vote(s) on reform or research based method shows 75% 
support: 


5.3 Interviewees/new teachers voice strong support or enthusiasm: 


5.4 Fewer than two interviewees/teachers voice dissent or indicate lack of 
use: 


5.5 Faculty maintains or increases support for method over time: 
Total Possible Score for Component 5 5 


* yes=1 and no=0 
(Continued) 
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Component Measure Score* 


6. Support for Teachers, Administrators, and Staff 


6.1 District supports model implementation: yes no 1 
6.2 District provides resources to support research-based method: yes no 1 
Total Possible Score for Component 6 2 


7. Parent and Community Involvement 
7.1 Emergence of new forms of parent involvement during CSR years: 
7.1.1 Special parent events 
7.1.2 Programs or opportunities for parents in instructional roles 
7.1.3. Parent advisory or other committees 


7.2 Level of parental involvement (high, medium, or low, defined as 
follows): 

high: you’ve observed parents in the school and interviewees voice strong high high=2 
or satisfactory level of parental involvement in school activities 

medium: school get traditional level of parental involvement (e.g., 10% 

attendance) medium medium=1 

low: no evidence of parental involvement beyond a handful of parents and 

interviewees voice low levels of participation low low=0 


7.3 Evidence of at least one community organization and one school/ 
community event or program: 


Total Possible Score for Component 7 4 
8. External Technical Support and Assistance 


8.1 Developer support and assistance (high, medium, or low, defined as 
follows): 

high: all3 CSR years plus during year after CSR high high=3 

medium: at least two of these four years medium medium=2 

/Jow: one or none of these four years low low=1 


8.2 Other external (but non-district) support and assistance: 
yes: evidence for a specific source and function on two or more 
occasions 
no: no such evidence (evidence can be documentation, interviewee 
mentions, or direct observation) 


Total Possible Score for Component 8 4 


* yes=1 and no=0 


(Continued) 
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Component Measure Score* 


9. Evaluation Strategies 
9.1 Existence of a written evaluation plan: 


9.2 Evidence of written evaluation findings (could even be a memo): 


9.3 Data are being used to suggest school enhancement: 


9.4 Data are used annually to improve instruction: 


Total Possible Score for Component 9 4 
10. Coordination of Resources 
10.1 Evidence of some coordination of funds from different external yes no 1 
sources (e.g., federal): 
10.2 Evidence of some coordination of external and local funds (i.e., core yes no l 
building): 
Total Possible Score for Component 10 2 
11. Evidence of Academic Improvement 
11.1 Annual student data indicates positive improvement in one or more yes no l 
areas: 
11.2 Student achievement data has increased incrementally since CSR yes no 1 
funding was received: 
Total Possible Score for Component 11 2 


Total Score 


* yes=1 and no=0 
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Field-based Component Site Visit Inventory for LACIO: 
11 Provisions of Comprehensive School Reform (CSR) 


A major purpose of the site visit is to observe a school’s progress in implementing the 11 provisions of CSR 
as specified by Sec. 1606 of P.L. 107-110 (No Child Left Behind Act of 2001). Your observations should be based 
on direct field evidence, including observations of key documents, copies of which should be obtained for the 
project files. The needed observations are organized according to the 11 components and are as follows. 


1. Employs Proven Strategies and Methods for Student Learning, Teaching, and School Management; 
combined with 


11. Has Support Based on: A) scientifically-based research, or B) strong evidence: 


1.1 Identify the proven method or strategy being used by the school (observe the method in action or examine 
samples of students’ work or other evidence showing how the method is being used). 

1.2 From documentation about the method, cite the following: 

1.1.1 The scientifically-based or other research evidence; 
1.1.2 The evidence of positive findings relative to comparison schools; 
1.1.3. The evidence of successful replication at other schools. 

1.2 Obtain or create a roster of classrooms that should be using the method; from this roster, select a random 
sample for classroom observation. Using the Classroom Observation Instrument (see separate instructions), estimate 
the percentage of classrooms using the method that should have been using it: 

1.3. Ask the developer or consultant providing external assistance to describe the school’s fidelity in 
implementing the method or strategy, and ask for the school’s summary rating, based on the scale that the developer 
or consultant uses. Determine that the developer or consultant has had no knowledge of student outcomes in making 
this rating. 

1.4 Score the extent of implementation of the method or strategy, using the adjusted Bodilly Scale, providing 
observational examples to support your score. 


2. Integrates a Comprehensive Design: 


2.1 Cite the plan, summary of faculty discussions, or other documentation, if any, that covers the full range of 
a school’s operations, and which therefore should embrace the school’s design for CSR (the documentation may be 
limited to CSR or go beyond it). 
2.2 From the documentation, cite the following: 
2.2.1 Needs assessment data; 
2.2.2 Quantitative performance goals; 
2.2.3 Discussion of specific curricula or instructional methods; 
2.2.4 Identification of assessment tools; 
2.2.5 Discussion of professional development; 
2.2.6 Indication of how financial resources will be strategically used. 
2.3 How many of the 11 CSR provisions can you find in the documentation? —__ 
2.4 Observe the salience of the design at the school (e.g., does anybody talk about it or its provisions?). 


3. Provides High Quality Professional Development: 


3.1 Cite the formal requirements for PD at this school (also check the district for PD requirements). 
3.2 Cite whether these requirements: 
3.2.1 Limit the substantively relevant topics eligible for PD; 
3.2.2 Discuss any preferred alignment between a) PD and b) specific 
standards, curriculum, or assessment tools: 
3.2.3 Specify the required number of PD days, per year: _. (Adjust 
the preceding estimate to exclude days for traditional teacher set-up 
in the fall and teacher clean-up in the spring). 
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3.3 Observe or obtain evidence of PD related to the CSR method or strategy, and review records of the extent 
of this PD (total number of teacher-hours) during the past year or semester. 
3.4 Observe or obtain evidence of a// other PD, and review records of the extent of this PD (total number of 
teacher-hours) during the past year or semester. 
3.5 Review PD records and determine the extent to which the PD: 
3.5.1 Has been content-focused and content-based 
3.5.2 Provides for coherent and continuous development 
3.5.3 Is classroom-based (e.g., assistance occurring in the classrooms, not 
at workshops) 
3.6 Collect evidence on the extent to which the school has used coaches, mentors, lead teachers, or other 
ongoing forms of assistance to teachers 


4. Includes Measurable Goals and Benchmarks for Student Academic Achievement: 


4.1 Cite the school’s documentation for goals (endpoints) and benchmarks (intermediate points). 
4.2 From the documentation, cite whether these goals and benchmarks cover: 
4.2.1 Student achievement, not just intermediate or other outcomes 
(e.g., attendance); 
4.2.2 Implementation variables, and not just outcome variables; 
4.2.3 The full range of grades at the school; 
4.2.4 All the core academic subjects at the school. 
4.3 Observe the salience of the goals and benchmarks at the school (e.g., are there explicit reward systems for 
accomplishing the goals and benchmarks; does anybody talk about them; are there posters or wallcharts about 
them?). 


5. Is Supported by School Staff: 


5.1 Cite the dates and results of any formal faculty vote(s) on CSR or on the research-based method. 

5.2 Cite whether at least 75% of interviewed staff members cite support for CSR or the research-based 
method; ditto dissent (at least 25%); ditto lack of use (at least 25%). 

5.5 Cite any other observations illustrating support for reform or for the research-based method (e.g., are 
there wall posters, banners, or other readily displayed materials referring to CSR?). 


6. Provides Support for School Staff: 


6.1 Cite ways in which the school supports the staff in the reform effort: 

6.1.1 Re-arrangement of school or teachers’ schedules 

6.1.2 Funds to teachers for supplemental classroom materials 

6.1.3 Salary or other formal credit for professional development specifically 
related to reform activities 

6.1.4 Salary for attending summer or weekend professional development 

6.1.5 Establishment of new positions (e.g., reform or parent coordinator) 
related to reform 

6.1.6 Other 


7. Provides for Meaningful Parent and Community Involvement (and is consistent with Sec. 1118 of P.L. 
107-110): 


7.1 Cite any written parent involvement policy. 

7.2 Observe or obtain evidence of parents’ role in their children’s learning (e.g., do you see parents in 
classrooms or otherwise assisting at the school? Is there any indication that parental support for children’s 
homework has changed?). 
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7.3 Observe or obtain evidence about parent-teacher communication (e.g., the frequency of parent-teacher 
conferences reflected by sign-in logs). 

7.4 Observe or obtain evidence about parent involvement in planning, implementing, or evaluating reform 

7.5 Corroborate items 7.2, 7.3, and 7.4 with events referenced in discussions with parents and teachers, also 
indicating how these parents and teachers were selected. 


8. Uses High Quality External Technical Support and Assistance: 


8.1 Identify the source(s) of external technical support and assistance, and review records indicating the 
frequency of such assistance over the past year or semester. 

8.2 Ascertain the principal’s satisfaction with the quality of the external support and assistance, as well as the 
reasons for this level of satisfaction. 

8.3 Observe a technical assistance event (e.g., workgroup or in-class session), reviewing any materials (e.g., 
agendas, exercises, hands-on materials) actually used in the event. Comment on teachers’ reactions and responses 
during the event: 

8.3.1 Teachers’ understanding of the substantive topics; 
8.3.2 Teachers’ motivation and support in participating in the event. 

8.4. Determine the annual amount of funds, if any, paid for the external assistance: _ 

8.5 Obtain documentation about the expertise of the external provider, in relation to experience with 
comprehensive school reform or research-based methods 


9. Includes Plan for Annual Evaluation of Implementation and Student Results: 


9.1 Cite the written evaluation plan. 
9.2 Review the plan for its coverage of implementation and student outcomes. 
9.3 Cite any observations or evidence showing that evaluation findings have been used. 


10. Identifies Other Resources to Coordinate and Sustain Reform: 


10.1 Indicate the resources (cite specific amounts and fiscal years), other than CSR, being used to coordinate 
or sustain reform, as cited in any budget, comprehensive plan, or other similar document. 

10.2 In what way and to what extent does the documentation indicate that coordination is occurring (e.g., a 
single plan, but different sources identified to support different subtasks)? 


List of School-Level Process Influences 
that Help a School to Implement CSR 


The list below deliberately ignores endemic conditions such as whether budgets are being cutback, staff 
turnover, student turnover, and other conditions that can impede change. The reason for ignoring these conditions is 
that the research question should be whether reform-oriented activities are better than non-reform-oriented activities, 
given the same endemic conditions. 


The list also ignores state or district policies because of some of the same reasoning and because they will be 
covered elsewhere in the research. 


A. An ongoing performance gap, either real or perceived: 
- school wants to be the best it can be (e.g., embraces an all students 
can learn vision) 
- school has not scored well in accountability systems 
- other external pressure to perform (e.g., busing or effects of 
desegregation court order) 
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Field Question: /dentify and cite the actual data demonstrating the existence or size of any performance gap, 
including the period of years covered. 


B. A principal with instructional and administrative expertise: 
- can serve as instructional leader 
- can create needed flexibility, adaptiveness, and resources to support change 
- can motivate staff 
Field Question: Jdentify and document the three most important actions taken by the principal to support 
comprehensive school reform. 


C. A match between the research-based method and the school’s needs (e.g., Stringfield and Datnow): 
- availability of some type of needs assessment (can be informal) 
- an informed process whereby alternative methods were considered, 
and the school understood well the grounds for making its final 
choice of methods 
Field Question: Document the process whereby such a match may or may not have taken place. 


D. Sufficient slack resources to support change (CSR award helps a lot if used properly; must then convert 
Title I and other funds later on, to maintain needed slack). 

Field Question: /dentify three incidences whereby slack resources were needed and determine whether they 
were made available. 


E. Sufficient professional development days and other staff resources (e.g., teacher substitutes) to permit 
teachers to obtain needed training and knowledge. 
Field Question: What arrangements are made for professional development? 


F. Analytic ability to connect school operations with student outcomes (e.g., district or external assisters may 
help to define relevant portions of the curriculum to be covered when a student underperforms some subtest). 
Field Question: Cite specific occasions when student performance is connected to needed curriculum 
work. 
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ADJUSTMENT TO THE BODILLY SCALE 
(see Appendix 4-1) 


Although the Bodilly Scale is was specifically designed to describe the level of 
observed implementation in a CSR school, during the field-focused study of CSRD schools 
it was determined that the definition of scale points was limited. While anchor points 0-3 
primarily focus on the early stages of implementation, the scale utilizes only two anchors 
(4-5) to describe the richness and variability of the implementation process. Therefore, 
in order to capture the variance within the implementation stages, the Bodilly Scale has 
been modified to include one additional anchored rating. 


Original Bodilly Scale 


0 — Not Implementing. No evidence of the element. 

1 — Planning. The school was planning or preparing to implement. 

2 — Piloting. The element was being partially implemented with only a small 
group of teachers or students involved. 

3 — Implementing. The majority of teachers were implementing the element, 
and the element was more fully developed in accordance with descriptions by 
the team. 

4 — Fulfilling. The element was evident across the school and was fully 
developed in accordance with the design teams’ descriptions. Signs of 
institutionalization were evident. 


Adjusted Bodilly Scale 


0 — Not Implementing. No evidence of the element. 

| — Planning. The school was planning or preparing to implement. 

2 — Piloting. The research-based model or practice was initially being 
implemented with only a small group of teachers or students involved with a 
plan for scale-up in future semesters or years. 

3 — Partially Implementing. The school is having difficulty maintaining the 
level of teacher or administrator buy-in necessary for full implementation, or is 
struggling with implementation for some other reason, resulting in only portions 
of the research-based model being implemented, or only portions fo teachers 
participating. 

4 — Implementing. The majority of teachers were implementing the majority 
fo aspects of a research-based model or practice. 

5 — Fulfilling. The research-based model or practice was evident across the 
school and was fully developed in accordance with the design teams’ 
descriptions. Signs of institutionalization were evident. 
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Classroom Observation Form 
For the Longitudinal Assessment of CSRD Implementation and Outcomes 


Instructions 


Please complete the attached Classroom Observation Form during the 2003 field-based 
site visits. The classroom observation form is designed to guide your field-based data collection. 


" Prior to the observation period, researchers should discuss what they expect 
to see in the classroom within the context of the reform method. While the 
categories may not exactly match what is happening in the classroom, the 
observer should attempt to classify the activities as best possible, and should use 
the comments section to clarify. 


" There is a possibility that researchers may not see the reform-related 
instructional strategies or techniques during every classroom visit. For 
example, although teachers may frequently engage in team teaching or modeling, 
observers may not have the opportunity to witness method-related techniques 
during the short observation period. 


" The classroom observations are just one source of evidence that may provide 
concrete examples or details the site visit teams can use in their reports. 


« There is space at the bottom of page one for observers to write brief 
descriptions of what is being observed. This section should include a brief 
description of how the observed activities might relate to the reform effort, the 
level of student engagement, and any other important items observed. Site visit 
team members also may need to ask classroom teachers to provide context for the 
lessons or activities that were observed. 


* In the section on the “Observed use of time,”’ observers can mark whether 
they observe the item in question at any time during the class period (as 
opposed to during each 10-minute block). If other major items or activities are 
taking place, or if clarification is necessary, the observer should provide these 
details in the comments section or on the back of this sheet. 


« In the section on the “Observed use of time,” Item 5 should be a percentage, 
reflecting the percent of time the majority of students are academically focused 
during the observation period, and Item 6 should be High, Medium or Low (High: 
Two or fewer students not participating; Medium: Less than 10 percent of 
students not participating, perhaps engaged in conversation, but not disruptive to 
majority of class; or Low: Many students doing their own thing). 
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Classroom Observation Form 
For the Longitudinal Assessment of CSRD Implementation and Outcomes 


Date: 

Time: 

School: Teacher‘(s): 
Model: 

Subject: 


Grade level: Observer: 


Implementation level, defined by school: 
Implementation level, defined by observer 
(to be completed following observation): 


Directions: 45-minute observation period, and 10 minute follow-up interview with 
teacher to check some items if necessary 


Observation Items 
A. Number of students in class: 


B. Number of students by racial/ethnic group: 
Black 
White 
Hispanic 
Asian 
Other 
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C. Observed use of time (Check in ten-minute blocks) 

Observers should complete this section for three 10-minute blocks within the 45-minute observation 
time. Observers should spend a short amount of time (5 minutes) familiarizing themselves with the 
classroom and curriculum prior to completing this section. Each ten-minute block may have more than 
one checkmark denoting the presence of an instruction strategy or method. (Comments are not 
necessary; can be used for individual observer’s notes). 


10-minute blocks 
Comments 
1. Instructional Orientation 
a. Direct instruction with the entire class 
(lecture or teacher question/student 
response) 
b. Team Teaching 
c. Cooperative/Collaborative Learning 
d. Individual Tutoring 
2. Classroom Organization 
a. Ability Grouping 
b. Multi-age Grouping 
c. Work centers (for individuals or groups) 
d. Individual desk arrangement 
3. Instructional Strategies 
a. Instructional feedback to enhance student 
learning 
. Integration of subject areas 
. Project-based learning 
. Use of higher-level thinking skills 
e. Teacher acting as coach/facilitator 
4. Student Activities 
a. Independent seat work 
b. Experiential, hands-on learning 
c. Systematic individual instruction 
(assignments geared to individual needs) 
d. Independent inquiry/research 
e. Student discussion 
5. Academically focused class time 
6. Level of student attention 


nao ot 


Adapted from “School Observation Form,” How to Evaluate Comprehensive School Reform 
Models, University of Memphis, developed by S. Ross for New American Schools, 2000. 
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D. EXAMPLES OF LINKAGES TO OTHER CSR COMPONENTS 

Observers should indicate the presence of these actions or materials in the classroom at any 
time during the 45-minute observation period. Other observations that relate to the 11 
components should be added to this form. 


Component \Observed | 
1 and 11. Proven (scientifically -based) strategies and methods for student learning; 
a. Teacher using curriculum developed or endorsed by the developer 
b. Teacher using instructional strategies called for by the method 
c. Materials carrying message of method’s concepts are displayed in the room 
d. Other: 
2. Integrates a comprehensive design: 
a. Copy of the school improvement plan present in the classroom 
b. Teacher observed using method’s instructional strategies during all subjects 
c. Other: 
3. Provides high quality professional development: 
a. Copy of teacher’s individual professional development plan present in the classroom 
b. Coaches or classroom observers in the classroom 
c. Other: 
4. Includes measurable goals and benchmarks for student academic achievement: 
a. Charts or graphs measuring student performance displayed on classroom walls 
b. Other: 
Is supported by school staff: 
a. Teacher actively using model curriculum or instructional strategies 
b. Model materials posted throughout the room 
c. Listen for positive comments about the lesson and school 
d. Other: 
6. Provides support for school staff: 
a. Class schedule allows teachers sufficient planning and meeting time 
b. Classrooms well equipped with instructional materials 
c. Other: 
7. Provides for the meaningful involvement of parents and the local community 
a. Parents or community volunteers present/assisting in the classroom 
b. Evidence of parent involvement in homework activities 
c. Other: 
8. Uses high quality external technical support and assistance 
a. Presence of conference or training materials in classroom 
b. Other: 
9, Includes plan for annual evaluation of the implementation and the student results; 
a. Other: 
10. Identifies other resources to coordinate and sustain reform 
a. Other: 


nm 
. 
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E. FOLLOW-UP INTERVIEW WITH THE TEACHER 
Choose two or three questions per class to ask each teacher following the observation. 
Observers should have answers to all questions by the end of all observations. 


1. Do you believe you receive adequate support from the school in order to implement the reform method? (Probe 
for examples.) 


2. Do you think there is alignment between the reform method, the professional development or technical assistance 
you receive, and the assessments that are administered to students? (Probe for examples.) 


3. Has the technical assistance you received met your needs and expectations? (Probe for examples.) 


4. Are there any reasons you don’t like to use the reform method, or have you experienced any problems in 
implementation? (Probe for examples, or reasons why other teachers might resist implementation.) 


5. How are parents involved in your students’ education? (Probe for classroom volunteers, homework assignments; 
ask for any documentation like sign-in sheets). 
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ADDENDUM TO SITE VISIT PROTOCOLS 


1. Add Component 11 to the Site Visit Protocol: 
“Scientifically-based research, or strong evidence that the school’s reform program will 
significantly improve academic performance” [note difference between Components | and 11]. 


2. Add “alignment” to Topics for District and State Officials: Inquire specifically about the 
how the district or state know about the alignment among standards, assessments, and curriculum 
and instruction—both within the state or district’s policies and between the district and state. 


3. Clarify the use of the Classroom Observation Form as follows: 

a. Randomly select classrooms from a classroom roster. 

b. Prior to the classroom observations, identify five instructional characteristics that 
illustrate the use of the method(s). Note those characteristics observed in every classroom, an 
indicator of the extent of implementation. 

c. If the method is content-specific, observe some classrooms that are illustrative of the 
method and others that are not. If the method is a comprehensive process model, then all 
classrooms are potentially illustrative of the model. Note on the form whether the classroom 
selected is method-specific or not. 

d. Note any other methods being implemented in the school, besides the CSR-related 
method. Most important, note if the “other methods” observed in the classroom are “add-ons” or 
integrated into a comprehensive whole. Make notes on the white space in the Form and discuss 
the issue in the case study report. 

e. Remember to enter the school code and teacher’s name on each page of the classroom 
observation form, to aid tracking after the site visit. Also, note the name of the CSR method and 
other methods used in the classroom. Other handwritten comments are encouraged. 


4. Watch for the incidence and potential effects of “student transfers,” 
as part of covering public school options 
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SCHOOL-LEVEL LOOSE ENDS 
FOR FIELD STUDY TEAMS FOR 2004-05 


Topics 


1. Was the CSR grant scaled-up or was it implemented all at once at the beginning of the grant 
period? Variability in start-ups, Timing for CSR. 


2. Where are the funds and who controls the funds? (Survey) That data could be collected during site 
visits in the principal and district interviews. “Where are the funds?” “Who makes decisions about 
how the CSR and other funds are used? [this could be addressed on the one-page principal process 
influence document or in principal’s interview] 


3. Burkett mentioned Schoolwide Title I (in Fed Register on July 2, 2004) Field staff may need to 
know about it before going to the field. 


4. Scope of Change (Maggie)—How demanding is the intervention, whether comprehensive or 
subject-based? Could be included in Component | (Research-based Method) 


5. If principal change, does principal deviate from adopted method or facilitate CSR grant? (Principal 
behavior, not whether a change occurred or not. Burkett) 


6. How to define PD? (Stephanie concerned about survey) SV protocol is complete. Might add across 
and among grade planning and training periods and structural PD, such as ongoing small learning 
communities to protocol. Expand definition of professional development (PD): The 
inventory/protocol addresses a wide variety of types of PD-school level. Site visitors should be 
reminded of the variety of topics that could be considered in PD, such as the number of hours spent in 
vertical and horizontal teacher planning time. 


7. Effects of Voluntary Public School Choice (VPSC): Describe student mobility in your school. 
(Probe: Has VPSC been a factor in student mobility?) 


8. Burkett, “Ask teachers if they participate in decisions.” [Do in the Leadership Team meeting in the 
prototype field study agenda. ] 


9. How are CSR funds being spent? 


10. Has this school been identified as low-performing? If so, has it been sanctioned because of low 
performance? 


11. Did the faculty formally vote to adopt the current reform in your school? 
12. What opportunities do teachers have for making management decisions? 
13. Do teachers have a common planning period in the day? 


14. Do any of the following occur in ;your school? (Probe: Looping, grouping by houses or families; 
student groups stay with teachers for more than one year; interdisciplinary teaching, and paired or 
teacher teaming) 


15. If a secondary school, what kind of student groups exist to achieve particular instructional goals? 


16. Who organizes the scope and sequence of the curriculum at this school? 


Protocol 
Item 
to be 
Referenced 


1.2 


10.1 


3.5 


School 
Operations 


School 
Operations 


B 
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TOPICS FOR DISTRICT AND STATE OFFICIALS 
FOR FIELD STUDY TEAMS, 2004-05 


District CSR Contact: 


1. Administration of CSR Program as a Whole: Define and update district involvement in 
CSR program (targeting and selection process, number of awardees, funding process, and goals 
for how whole-school improvement will be achieved). 
2. Technical Assistance and Implementation Support: Describe state technical assistance 
and other support provided to districts and schools that might be related to CSR. 
3. Relationships between CSR and District Initiatives. Describe the relationships, if any, 
between CSR and: 
a) district reform initiatives (e.g., standards, curriculum requirements, use of 
research-based methods, new report card forms) 
b) district administration of Title I and schoolwide Title I 
c) district requirements for PD 
4. Relationships between CSR and NCLB and Other Federal and State Initiatives. 
Describe the relationships, if any, between CSR and other initiatives, such as NCLB and public 
school choice (federal) and state accountability and reform (state)—e.g., any relationships 
between CSR and state standards and assessments, and the implications for CSR implementation. 
5. Monitoring and Evaluation: Describe whether or how the district monitors and 
evaluates CSR implementation and outcomes—if possible focusing specifically on the two site 
visit schools. 


State CSR Contact: 


1. Administration of CSR Program as a Whole: Define and update state involvement in 
CSR program (targeting and selection process, number of awardees, funding process, and goals 
for how whole-school improvement will be achieved). Describe state priorities for CSR. 
2. Technical Assistance and Implementation Support: Describe state technical assistance 
and other support provided to districts and schools that might be related to CSR. 
3. Relationships between CSR and State Initiatives. Describe the relationships, if any, 
between CSR and: 
a) state reform initiatives (e.g., standards, curriculum requirements, use of 
research-based methods) 
b) state assessments and accountability initiatives 
c) state requirements for PD, teacher certification, or related policies 
4. Relationships between CSR and NCLB and Other Federal Initiatives. Describe the 
relationships, if any, between CSR and other federal initiatives, such as NCLB and public school 
choice. 
5. Monitoring and Evaluation: Describe whether 0 how the state monitors and evaluates 
CSR implementation and outcomes—if possible focusing specifically on the two site visit 
schools. Obtain a copy of the state CSR evaluation report (there may be several such reports, 
each completed annually). 
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Field-based Component Site Visit Inventory for LACIO: 
11 Provisions of Comprehensive School Reform (CSR) 


A major purpose of the site visit is to observe a school’s progress in implementing the 11 provisions of CSR 
as specified by Sec. 1606 of P.L. 107-110 (No Child Left Behind Act of 2001). Your observations should be based 
on direct field evidence, including observations of key documents, copies of which should be obtained for the 
project files. The needed observations are organized according to the 11 components and are as follows. 


1. Employs Proven Strategies and Methods for Student Learning, Teaching, and School Management; 
combined with 
11. Has Support Based on: A) scientifically-based research, or B) strong evidence: 


1.1 Identify the proven method or strategy being used by the school (observe the method in action or examine 
samples of students’ work or other evidence showing how the method is being used). 

1.2 From documentation about the method, cite the following: 

1.1.1 The scientifically-based or other research evidence; 
1.1.2 The evidence of positive findings relative to comparison schools; 
1.1.3 The evidence of successful replication at other schools. 

1.2 Obtain or create a roster of classrooms that should be using the method; from this roster, select a random 
sample for classroom observation. Using the Classroom Observation Instrument (see separate instructions), estimate 
the percentage of classrooms using the method that should have been using it: 

1.3. Ask the developer or consultant providing external assistance to describe the school’s fidelity in 
implementing the method or strategy, and ask for the school’s summary rating, based on the scale that the developer 
or consultant uses. Determine that the developer or consultant has had no knowledge of student outcomes in making 
this rating. 

1.4 Score the extent of implementation of the method or strategy, using the adjusted Bodilly Scale, providing 
observational examples to support your score. 


2. Integrates a Comprehensive Design: 


2.1 Cite the plan, summary of faculty discussions, or other documentation, if any, that covers the full range of 
a school’s operations, and which therefore should embrace the school’s design for CSR (the documentation may be 
limited to CSR or go beyond it). 
2.2 From the documentation, cite the following: 
2.2.1 Needs assessment data; 
2.2.2 Quantitative performance goals; 
2.2.3 Discussion of specific curricula or instructional methods; 
2.2.4 Identification of assessment tools: 
2.2.5 Discussion of professional development; 
2.2.6 Indication of how financial resources will be strategically used. 
2.3 How many of the 11 CSR provisions can you find in the documentation? —__ 
2.4 Observe the salience of the design at the school (e.g., does anybody talk about it or its provisions?). 


3. Provides High Quality Professional Development: 


3.1 Cite the formal requirements for PD at this school (also check the district for PD requirements). 
3.2 Cite whether these requirements: 
3.2.1 Limit the substantively relevant topics eligible for PD; 
3.2.2 Discuss any preferred alignment between a) PD and b) specific 
standards, curriculum, or assessment tools: 
3.2.3 Specify the required number of PD days, per year: _. (Adjust 
the preceding estimate to exclude days for traditional teacher set-up 
in the fall and teacher clean-up in the spring). 
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3.3 Observe or obtain evidence of PD related to the CSR method or strategy, and review records of the extent 
of this PD (total number of teacher-hours) during the past year or semester. 
3.4 Observe or obtain evidence of all other PD, and review records of the extent of this PD (total number of 
teacher-hours) during the past year or semester. 
3.5 Review PD records and determine the extent to which the PD: 
3.5.1 Has been content-focused and content-based 
3.5.2 Provides for coherent and continuous development 
3.5.3 Is classroom-based (e.g., assistance occurring in the classrooms, not 
at workshops) 
3.6 Collect evidence on the extent to which the school has used coaches, mentors, lead teachers, or other 
ongoing forms of assistance to teachers 


4. Includes Measurable Goals and Benchmarks for Student Academic Achievement: 


4.1 Cite the school’s documentation for goals (endpoints) and benchmarks (intermediate points). 
4.2 From the documentation, cite whether these goals and benchmarks cover: 
4.2.1 Student achievement, not just intermediate or other outcomes 
(e.g., attendance); 
4.2.2 Implementation variables, and not just outcome variables; 
4.2.3 The full range of grades at the school; 
4.2.4 All the core academic subjects at the school. 
4.3 Observe the salience of the goals and benchmarks at the school (e.g., are there explicit reward systems for 
accomplishing the goals and benchmarks; does anybody talk about them; are there posters or wallcharts about 
them?). 


5. Is Supported by School Staff: 


5.1 Cite the dates and results of any formal faculty vote(s) on CSR or on the research-based method. 

5.2 Cite whether at least 75% of interviewed staff members cite support for CSR or the research-based 
method; ditto dissent (at least 25%); ditto lack of use (at least 25%). 

5.5 Cite any other observations illustrating support for reform or for the research-based method (e.g., are 
there wall posters, banners, or other readily displayed materials referring to CSR?). 


6. Provides Support for School Staff: 


6.1 Cite ways in which the school supports the staff in the reform effort: 

6.1.1 Re-arrangement of school or teachers’ schedules 

6.1.2 Funds to teachers for supplemental classroom materials 

6.1.3 Salary or other formal credit for professional development specifically 
related to reform activities 

6.1.4 Salary for attending summer or weekend professional development 

6.1.5 Establishment of new positions (e.g., reform or parent coordinator) 
related to reform 

6.1.6 Other 


7. Provides for Meaningful Parent and Community Involvement (and is consistent with Sec. 1118 of P.L. 
107-110): 


7.1 Cite any written parent involvement policy. 

7.2 Observe or obtain evidence of parents’ role in their children’s learning (e.g., do you see parents in 
classrooms or otherwise assisting at the school? Is there any indication that parental support for children’s 
homework has changed?). 
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7.3 Observe or obtain evidence about parent-teacher communication (e.g., the frequency of parent-teacher 
conferences reflected by sign-in logs). 

7.4 Observe or obtain evidence about parent involvement in planning, implementing, or evaluating reform 

7.5 Corroborate items 7.2, 7.3, and 7.4 with events referenced in discussions with parents and teachers, also 
indicating how these parents and teachers were selected. 


8. Uses High Quality External Technical Support and Assistance: 


8.1 Identify the source(s) of external technical support and assistance, and review records indicating the 
frequency of such assistance over the past year or semester. 

8.2 Ascertain the principal’s satisfaction with the quality of the external support and assistance, as well as the 
reasons for this level of satisfaction. 

8.3 Observe a technical assistance event (e.g., workgroup or in-class session), reviewing any materials (e.g., 
agendas, exercises, hands-on materials) actually used in the event. Comment on teachers’ reactions and responses 
during the event: 

8.3.1 Teachers’ understanding of the substantive topics; 
8.3.2 Teachers’ motivation and support in participating in the event. 

8.4. Determine the annual amount of funds, if any, paid for the external assistance: _ 

8.5 Obtain documentation about the expertise of the external provider, in relation to experience with 
comprehensive school reform or research-based methods 


9. Includes Plan for Annual Evaluation of Implementation and Student Results: 


9.1 Cite the written evaluation plan. 
9.2 Review the plan for its coverage of implementation and student outcomes. 
9.3 Cite any observations or evidence showing that evaluation findings have been used. 


10. Identifies Other Resources to Coordinate and Sustain Reform: 


10.1 Indicate the resources (cite specific amounts and fiscal years), other than CSR, being used to coordinate 
or sustain reform, as cited in any budget, comprehensive plan, or other similar document. 

10.2 In what way and to what extent does the documentation indicate that coordination is occurring (e.g., a 
single plan, but different sources identified to support different subtasks)? 


List of School-Level Process Influences 
that Help a School to Implement CSR 


The list below deliberately ignores endemic conditions such as whether budgets are being cutback, staff 
turnover, student turnover, and other conditions that can impede change. The reason for ignoring these conditions is 
that the research question should be whether reform-oriented activities are better than non-reform-oriented activities, 
given the same endemic conditions. 


The list also ignores state or district policies because of some of the same reasoning and because they will be 
covered elsewhere in the research. 


A. An ongoing performance gap, either real or perceived: 
- school wants to be the best it can be (e.g., embraces an all students 
can learn vision) 
- school has not scored well in accountability systems 
- other external pressure to perform (e.g., busing or effects of 
desegregation court order) 
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